AI-based intelligent rendering method and system for UI layout information

Abstract

This invention relates to the field of UI layout rendering technology, and in particular to an AI-based intelligent rendering method and system for UI layout information, comprising: parsing UI design source files, extracting layout structure and visual attributes, generating an element spatial relationship diagram, performing visual appearance and spatial context encoding fusion on the nodes in the diagram to obtain a joint representation vector, inferring semantic categories based on the joint representation vector, constructing a structured layout tree, and converting the layout tree into native layout code according to the characteristics of the target rendering platform, thereby realizing automatic conversion from design source files to platform native code and improving the efficiency and accuracy of UI layout rendering.

Description

Technical Field

[0001] This invention relates to the field of UI layout rendering technology, and in particular to an AI-based intelligent rendering method and system for UI layout information. Background Technology

[0002] In the field of user interface design, existing UI layout rendering methods typically rely on designers manually coding or using static code exported from design tools. Designers arrange visual elements in the design draft, such as buttons, text boxes, and images, according to fixed coordinates and sizes, and implement the interface layout using platform-specific languages ​​such as CSS, Android XML, or iOS Storyboard. Some tools, such as Sketch and Figma, provide plugins that can generate basic code snippets, but these codes only retain the position and style attributes of elements, lacking an understanding of the spatial relationships and semantic categories between elements. Developers still need to manually adjust the layout structure according to the design intent, especially when dealing with nested levels, responsive adaptation, and cross-platform conversions, which is labor-intensive and prone to introducing errors.

[0003] One drawback of conventional approaches is the lack of automatic semantic recognition for UI elements. Existing methods only focus on visual attributes such as size, color, and font, failing to distinguish whether a rectangular area is a button or a card container. This results in all elements in the generated code being treated as ordinary views or images, unable to automatically associate corresponding interaction logic. Developers must check each element individually and manually assign the correct component type (such as Button, ListView), a time-consuming and error-prone process. Especially on complex interfaces, the lack of semantic tags can compromise code maintainability and reusability.

[0004] Another drawback is the difficulty in accurately capturing the spatial context and nesting relationships between elements. In design drafts, elements express hierarchy and subordination through visual patterns such as stacking, alignment, and grouping. Existing rendering solutions often simplify this to absolute positioning or linear arrangement, ignoring the constraints and hierarchical structure between adjacent elements. For example, a floating button might be incorrectly treated as an independent element at the same level as the background, rather than being located within a container. This causes misalignment in the generated layout when scaling or adapting, failing to reflect the designer's true intent. This loss of spatial information results in unstructured code, making it difficult to use directly in responsive or cross-platform development scenarios. Summary of the Invention

[0005] This invention provides an AI-based intelligent rendering method and system for UI layout information, which can solve the problems in the prior art.

[0006] A first aspect of this invention provides an AI-based intelligent rendering method for UI layout information, comprising: The user interface design source file is obtained and parsed to extract layout structure information and visual element attribute information. Based on the layout structure information, spatial relationship modeling is performed on each visual element to generate an element spatial relationship diagram. In the element spatial relationship diagram, nodes represent each visual element and edges represent the spatial adjacency and nesting hierarchy relationships between visual elements. The visual appearance features of each node in the element spatial relationship graph are encoded to generate a visual appearance feature vector. Based on the spatial context topology of each node in the element spatial relationship graph, a spatial context feature vector is generated. The visual appearance feature vector and the spatial context feature vector are fused to obtain a joint representation vector. Semantic category inference is performed based on joint representation vectors, and element type labels are output. Based on the element type labels and the element spatial relationship graph, a structured layout tree carrying semantic information is constructed. Each node of the structured layout tree contains element type labels, spatial location parameters and style attribute sets. Based on the rendering engine characteristics of the target rendering platform, the structured layout tree is mapped and converted into the native layout code of the target rendering platform. The native layout code includes platform native UI component declarations corresponding one-to-one with each node in the structured layout tree and the hierarchical nesting relationship between components.

[0007] Based on the layout structure information, spatial relationship modeling is performed on each visual element to generate an element spatial relationship diagram, including: Extract the bounding box coordinate parameters and document object model (DOM) hierarchical path of each visual element from the layout structure information. Based on the bounding box coordinate parameters of each visual element, calculate the horizontal and vertical bounding box adjacency of each visual element pairwise. Mark visual element pairs that satisfy the horizontal adjacency condition and the vertical adjacency condition as spatial adjacency. According to the hierarchical index identifier of each visual element in the DOM hierarchical path, mark visual element pairs that are in the same DOM hierarchical path and whose hierarchical index identifiers satisfy the parent-child index relationship as nested hierarchical relationships. Traverse all visual element pairs and construct the element spatial relationship graph with each visual element as a node. Establish spatially adjacent undirected edges between nodes corresponding to visual element pairs with spatial adjacency, and establish nested directed edges from parent nodes to child nodes between nodes corresponding to visual element pairs with nested hierarchical relationships. The bounding box coordinate parameters, document object model level depth value, and node adjacency degree of each node in the element spatial relationship graph are associated with the corresponding visual element. The node adjacency degree is the sum of the number of spatially adjacent undirected edges and the number of nested directed edges directly connected to the node.

[0008] Establish nested directed edges between parent and child nodes of visual element pairs with nested hierarchical relationships, including: For parent and child visual elements with a parent-child index relationship, a first nested directed edge pointing from the parent node to the child node and a second nested directed edge pointing from the child node to the parent node are respectively established in the element space relationship graph to form a parent-child bidirectional index structure. The first nested directed edge carries the relative position offset vector and relative size ratio vector of the child node in the coordinate system of the parent node, and the second nested directed edge carries the layout constraint type identifier and constraint reference direction applied by the parent node to the child node. Based on the bounding box coordinate parameters of the parent node visual element and the bounding box coordinate parameters of the child node visual element, the proportion of the area of ​​the child node bounding box contained inside the parent node bounding box to the total area of ​​the parent node bounding box is calculated, as well as the spacing offset of the four boundaries of the child node bounding box relative to the corresponding four boundaries of the parent node bounding box. The area proportion and the spacing offset of each boundary are encoded into the edge attribute vector of the first nested directed edge. The edge attribute vector is fused with the visual appearance feature vector of the corresponding node in the subsequent spatial context feature encoding process of the node to characterize the spatial constraint strength and boundary alignment relationship between the parent and child nodes.

[0009] The visual appearance features of each node in the element spatial relationship graph are encoded to generate a visual appearance feature vector. Based on the spatial context topology of each node in the element spatial relationship graph, a spatial context feature vector is generated, including: For each node, extract the following three types of visual appearance features from the corresponding visual elements: bounding box geometric features, including the aspect ratio, area, and tightness of the bounding box; color distribution statistical features, including the color histogram distribution vector and the dominant color identifier; and text content semantic features, including the text area ratio and the number of text lines. Concatenate the bounding box geometric features, the color distribution statistical features, and the text content semantic features into feature vectors to generate the visual appearance feature vector of the node. Based on the element spatial relationship graph, for each node, a set of spatially adjacent nodes directly connected to the node through undirected edges and a set of nested hierarchical associated nodes directly connected to the node through nested directed edges are collected. The visual appearance feature vectors of each adjacent node in the spatially adjacent node set are aggregated by mean to generate a spatially adjacent aggregated feature vector. The visual appearance feature vectors of the parent node and child node in the nested hierarchical associated node set are aggregated by mean to generate a hierarchical associated aggregated feature vector. The spatially adjacent aggregated feature vector and the hierarchical associated aggregated feature vector are concatenated to generate the spatial context feature vector of the node.

[0010] The visual appearance feature vectors of each neighboring node in the spatial neighboring node set are aggregated by mean to generate a spatial neighboring aggregated feature vector, including: Based on the degree of adjacency between the bounding boxes of the node and each of the adjacent nodes in the spatial neighboring node set, the spatial proximity weight factor of each adjacent node is calculated. The spatial proximity weight factor is obtained by weighted summation of the proportion of the adjacency area between the bounding box of the node and the bounding boxes of the adjacent nodes and the proportion of the boundary adjacency length. The value of the spatial proximity weight factor is positively correlated with the degree of adjacency between the bounding boxes of the node and the adjacent nodes. Based on the adjacency degree of each adjacent node in the spatial relationship graph of the elements, the topological centrality weight factor of each adjacent node is calculated. The topological centrality weight factor is determined by the normalized ratio of the adjacency degree of each adjacent node to the sum of the adjacency degrees of all adjacent nodes in the spatial adjacent node set. The value of the topological centrality weight factor is positively correlated with the adjacency degree of the node. Each feature dimension component in the visual appearance feature vector of each adjacent node is multiplied by the corresponding spatial proximity weight factor and topological centrality weight factor. The weighted visual appearance feature vectors of each adjacent node are summed dimension by dimension. The result of the summation dimension by dimension is divided by the number of nodes in the spatial adjacent node set to obtain the spatial adjacent aggregated feature vector.

[0011] Semantic category inference is performed based on joint representation vectors, and element type labels are output. A structured layout tree carrying semantic information is constructed based on the element type labels and the element spatial relationship graph, including: Based on the joint representation vector of each visual element, the probability value of each visual element belonging to each preset semantic category is calculated, and the preset semantic category corresponding to the maximum probability value is determined as the element type label of the visual element. The preset semantic categories include container, text, image, list and interactive control categories. The node corresponding to the visual element with the smallest document object model hierarchy depth value in the element spatial relationship graph is taken as the root node of the structured layout tree. The parent-child hierarchy relationship between each node is determined according to the direction of the nested directed edges in the element spatial relationship graph. Each node is inserted into the corresponding hierarchy position in the structured layout tree layer by layer according to the parent-child hierarchy relationship. Each inserted node carries the element type label, spatial position parameters and style attribute set of the corresponding visual element in the structured layout tree. Perform hierarchical semantic consistency verification on the structured layout tree. If there is a predefined semantic conflict pattern between the element type label of the target node and the element type label combination of each child node in the set of child nodes of the target node, then re-extract the spatial context feature vector of the target node, and re-infer the semantic category of the target node and update its element type label based on the re-extracted spatial context feature vector.

[0012] Based on the rendering engine characteristics of the target rendering platform, the structured layout tree is mapped and converted into the native layout code of the target rendering platform, including: Obtain the list of native UI component types, component nesting constraint rule set, and style attribute naming conventions supported by the rendering engine of the target rendering platform. Based on the functional description of each native UI component in the list of native UI component types and the semantic description of each element type tag in the structured layout tree, establish a mapping relationship table from element type tags to native UI component types. A depth-first traversal is performed on the structured layout tree, and each node is processed in the order of traversal. The corresponding target native UI component type is obtained by querying the mapping table based on the element type label of the currently traversed node. The platform native UI component declaration code corresponding to the node is generated based on the target native UI component type. According to the parent-child hierarchy relationship of the currently traversed node in the structured layout tree, the platform native UI component declaration code corresponding to the child node is nested and written into the inner area of ​​the platform native UI component declaration code corresponding to the parent node. Each style attribute name in the style attribute set of the currently traversed node is converted into the native style attribute name used by the target rendering platform according to the style attribute naming specification. The converted native style attribute name and the corresponding style attribute value are then associated with the platform native UI component declaration code corresponding to the node in the form of key-value pairs.

[0013] A second aspect of this invention provides an AI-based intelligent rendering system for UI layout information, comprising: The analytical modeling unit is used to obtain and parse the user interface design source file, extract layout structure information and visual element attribute information, and perform spatial relationship modeling on each visual element based on the layout structure information to generate an element spatial relationship diagram. In the element spatial relationship diagram, each visual element is represented by a node and the spatial adjacency relationship and nesting hierarchy relationship between visual elements are represented by an edge. The feature encoding unit is used to encode the visual appearance features of each node in the element spatial relationship graph to generate a visual appearance feature vector, and to encode the spatial context topology structure of each node in the element spatial relationship graph to generate a spatial context feature vector. The visual appearance feature vector and the spatial context feature vector are fused to obtain a joint representation vector. A semantic construction unit is used to perform semantic category inference based on joint representation vectors, output element type labels, and construct a structured layout tree carrying semantic information based on the element type labels and the element spatial relationship graph. Each node of the structured layout tree contains element type labels, spatial location parameters and style attribute sets. The code conversion unit is used to map the structured layout tree into the native layout code of the target rendering platform according to the rendering engine characteristics of the target rendering platform. The native layout code includes platform native UI component declarations and hierarchical nesting relationships between components that correspond one-to-one with each node in the structured layout tree.

[0014] A third aspect of the present invention provides an electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0015] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0016] This method significantly improves the automation and intelligence of UI layout rendering. By parsing the design source files and constructing an element spatial relationship graph, it accurately captures the spatial adjacency and nesting hierarchy relationships between visual elements, laying a structured foundation for subsequent semantic analysis. Utilizing multimodal feature fusion technology, it jointly encodes visual appearance features and spatial context features, greatly enhancing the accuracy of element semantic category inference. This effectively overcomes the limitations of traditional methods that rely on manual rules or template matching, and is particularly suitable for scenarios with ambiguous element types or diverse styles in complex interfaces.

[0017] Based on semantic tags inferred from joint representation vectors, and combined with a structured layout tree generated from a spatial relationship graph, the type, position, and style attribute information of each UI component are fully preserved. This hierarchical and semantic data organization makes the layout logic clear and traceable, facilitating later debugging and maintenance. Compared to directly parsing and generating flat code, this method avoids component hierarchy chaos and missing styles, significantly reduces visual deviations caused by differences in rendering engines, and improves the stability of cross-platform UI reuse.

[0018] During the target rendering platform adaptation phase, a seamless conversion from design drafts to executable code is achieved by mapping the structured layout tree to native UI component declarations and nesting relationships. This method eliminates the need for manual adaptation code, automatically adapting to the characteristics of different rendering engines and significantly shortening the multi-platform UI development cycle. Simultaneously, the efficiency and platform compatibility of the native code ensure that the final rendering effect is highly consistent with the design draft, resulting in a significantly improved user experience. Overall, this invention provides an end-to-end intelligent solution for UI development, balancing accuracy, efficiency, and cross-platform capabilities. Attached Figure Description

[0019] Figure 1 A flowchart illustrating the AI-based intelligent rendering method for UI layout information. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0022] Figure 1 This is a flowchart illustrating the AI-based intelligent rendering method for UI layout information according to an embodiment of the present invention.

[0023] AI-based intelligent rendering methods for UI layout information include: The user interface design source file is obtained and parsed to extract layout structure information and visual element attribute information. Based on the layout structure information, spatial relationship modeling is performed on each visual element to generate an element spatial relationship diagram. In the element spatial relationship diagram, nodes represent each visual element and edges represent the spatial adjacency and nesting hierarchy relationships between visual elements. The visual appearance features of each node in the element spatial relationship graph are encoded to generate a visual appearance feature vector. Based on the spatial context topology of each node in the element spatial relationship graph, a spatial context feature vector is generated. The visual appearance feature vector and the spatial context feature vector are fused to obtain a joint representation vector. Semantic category inference is performed based on joint representation vectors, and element type labels are output. Based on the element type labels and the element spatial relationship graph, a structured layout tree carrying semantic information is constructed. Each node of the structured layout tree contains element type labels, spatial location parameters and style attribute sets. Based on the rendering engine characteristics of the target rendering platform, the structured layout tree is mapped and converted into the native layout code of the target rendering platform. The native layout code includes platform native UI component declarations corresponding one-to-one with each node in the structured layout tree and the hierarchical nesting relationship between components.

[0024] Based on the layout structure information, spatial relationship modeling is performed on each visual element to generate an element spatial relationship diagram, including: Extract the bounding box coordinate parameters and document object model (DOM) hierarchical path of each visual element from the layout structure information. Based on the bounding box coordinate parameters of each visual element, calculate the horizontal and vertical bounding box adjacency of each visual element pairwise. Mark visual element pairs that satisfy the horizontal adjacency condition and the vertical adjacency condition as spatial adjacency. According to the hierarchical index identifier of each visual element in the DOM hierarchical path, mark visual element pairs that are in the same DOM hierarchical path and whose hierarchical index identifiers satisfy the parent-child index relationship as nested hierarchical relationships. Traverse all visual element pairs and construct the element spatial relationship graph with each visual element as a node. Establish spatially adjacent undirected edges between nodes corresponding to visual element pairs with spatial adjacency, and establish nested directed edges from parent nodes to child nodes between nodes corresponding to visual element pairs with nested hierarchical relationships. The bounding box coordinate parameters, document object model level depth value, and node adjacency degree of each node in the element spatial relationship graph are associated with the corresponding visual element. The node adjacency degree is the sum of the number of spatially adjacent undirected edges and the number of nested directed edges directly connected to the node.

[0025] The layout structure information extracted from the parsed user interface design source file contains a detailed structured description of each visual element in the document. This information covers not only the precise spatial position of the elements in the canvas or viewport, but also their organization within the document object model. Visual elements can be various UI components such as buttons, text boxes, image containers, navigation bars, and card components, each with clearly defined boundaries in the design source file.

[0026] When extracting the bounding box coordinates of each visual element, the coordinates of the top-left and bottom-right corners of the bounding box of each visual element are obtained, or a representation using the top-left corner coordinates combined with width and height is used. For example, if the top-left corner coordinates of a button element's bounding box are (120 pixels, 200 pixels), its width is 80 pixels, and its height is 40 pixels, then the complete bounding box coordinates of this element are recorded as: starting x-coordinate 120, starting y-coordinate 200, ending x-coordinate 200, and ending y-coordinate 240. This coordinate parameter can accurately describe the spatial area occupied by the element on a two-dimensional plane.

[0027] The Document Object Model (DOM) hierarchy path describes the position of a visual element within the entire UI structure tree, expressed as a path string or a sequence of hierarchical indices. For example, if a text label is the first list item in the second panel container under the root container, its DOM hierarchy path can be represented as " / root / panel[2] / list-item[1] / text-label", where each hierarchical index indicates the element's sequential position within its parent container. By parsing such a hierarchy path, the subordinate relationships of elements in the nested structure can be clearly reconstructed.

[0028] When calculating the horizontal adjacency of bounding boxes, for any two visual elements' bounding boxes, the first step is to determine if they have projection overlap in the vertical direction. Specifically, the ordinate ranges of the two bounding boxes are compared. If the starting ordinate of the first bounding box is less than the ending ordinate of the second bounding box, and the ending ordinate of the first bounding box is greater than the starting ordinate of the second bounding box, then there is a projection overlap area between them in the vertical direction. Given this vertical projection overlap, the shortest horizontal distance between the two bounding boxes is further calculated. If the first bounding box is to the left of the second bounding box, the horizontal distance is the starting x-coordinate of the second bounding box minus the ending x-coordinate of the first bounding box; if the first bounding box is to the right, the horizontal distance is the starting x-coordinate of the first bounding box minus the ending x-coordinate of the second bounding box. This horizontal distance is normalized by dividing it by the average width of the two bounding boxes or the total width of the canvas. The resulting value is the metric for the horizontal adjacency of the bounding boxes. When this metric is less than a preset horizontal adjacency threshold (e.g., 0.05 or 5 pixels), the two elements are considered to meet the horizontal adjacency condition.

[0029] The vertical adjacency of bounding boxes is calculated using a symmetrical logic. First, it's determined whether the two bounding boxes have horizontal projection overlap, meaning the starting x-coordinate of the first bounding box is less than the ending x-coordinate of the second bounding box, and the ending x-coordinate of the first bounding box is greater than the starting x-coordinate of the second bounding box. If horizontal projection overlap exists, the shortest vertical distance between the two bounding boxes is calculated. If the first bounding box is above the second bounding box, the vertical distance is the starting y-coordinate of the second bounding box minus the ending y-coordinate of the first bounding box; if it's below, it's the starting y-coordinate of the first bounding box minus the ending y-coordinate of the second bounding box. This distance is then normalized and compared to a preset vertical adjacency threshold to determine if the vertical adjacency condition is met.

[0030] When a pair of visual elements simultaneously meet the criteria of horizontal adjacency (both horizontally and vertically), they are considered spatially adjacent. For example, in a form interface, an input box label and the input text box immediately to its right are roughly aligned vertically (high vertical adjacency) and very close horizontally (high horizontal adjacency), thus they are identified as spatially adjacent. This adjacency reflects the physical arrangement of visual elements in the interface layout.

[0031] When identifying nested hierarchical relationships based on the hierarchical path of the Document Object Model, the hierarchical path string of each visual element is parsed, and the hierarchical index sequence is extracted. Assuming that the hierarchical path of element A is " / root / container[1] / group[3]" and the hierarchical path of element B is " / root / container[1] / group[3] / button[2]", by comparison, it can be found that the hierarchical path of element B is prefixed with the hierarchical path of element A, and the hierarchical depth of element B is one more level than that of element A. In this case, element A is the parent node and element B is the child node, and the two constitute a nested hierarchical relationship. The numerical sequence in the hierarchical index identifier reflects the sibling order of elements under the same parent container. By judging the inclusion relationship of the hierarchical path and the difference in hierarchical depth, the parent-child nested relationship can be accurately identified.

[0032] When constructing the element spatial relationship graph, first, a set of nodes equal to the number of visual elements is created, with each node corresponding to one visual element, and a unique identifier assigned to each node. All combinations of visual element pairs are traversed, and for each pair, it is checked whether it satisfies the aforementioned spatial adjacency or nested hierarchical relationship criteria. For element pairs marked as spatially adjacent, an undirected edge is established between the corresponding two nodes in the graph structure. The existence of this edge indicates that the two visual elements are closely adjacent in spatial layout; this edge is not direction-sensitive because the adjacency relationship itself is symmetrical. For element pairs marked as nested hierarchical, a directed edge is established in the graph structure in the direction from the parent node to the child node. The directionality of this edge clearly indicates the hierarchical relationship of inclusion and being included. This directed edge constitutes a tree-like substructure in the element spatial relationship graph.

[0033] For each node in the graph, the bounding box coordinate parameters of the corresponding visual element are associated. These parameters include the starting x-coordinate, starting y-coordinate, ending x-coordinate, and ending y-coordinate of the bounding box, or stored as center point coordinates combined with width and height. This coordinate information serves as the spatial position attribute of the node, providing basic data for subsequent visual appearance feature encoding. When calculating the document object model (DOM) hierarchy depth value, the nesting level of the visual element in the DOM hierarchy path is counted. The hierarchy depth value of the root node is 0, the hierarchy depth value of its direct child nodes is 1, and so on. The hierarchy depth value reflects the nesting depth of the element in the UI structure; deeply nested elements often have more specific semantic roles.

[0034] When calculating the adjacency degree of a node, the number of all edges directly connected to that node is counted, including spatially adjacent undirected edges and nested directed edges. For undirected edges, if a node has a spatially adjacent undirected edge with another node, that edge contributes 1 to the node's adjacency degree. For directed edges, regardless of whether the node is a parent or child node, each nested directed edge connected to that node contributes 1 to its adjacency degree. For example, if a container node contains 3 child elements, there are 3 nested directed edges emanating from that container node; simultaneously, this container node may have 2 spatially adjacent undirected edges with other sibling container nodes, so its adjacency degree is 5. The node's adjacency degree reflects the connection complexity of that visual element in the layout; nodes with higher degrees often play a key organizing or pivotal role in the interface structure.

[0035] In practical applications, for a mobile application interface that includes a navigation bar, content area, and bottom toolbar, the navigation bar node and content area node are connected by spatially adjacent undirected edges because they are closely arranged vertically. The navigation bar node itself acts as a parent node, connecting its internal title text node, back button node, and menu icon node through nested directed edges. This spatial relationship graph of elements can fully capture the spatial layout topology and hierarchical organization structure of the interface, providing structured input for subsequent graph neural network feature extraction.

[0036] Through the aforementioned spatial relationship modeling process, the originally flat set of visual elements is transformed into a graph structure with rich spatial and hierarchical relationships. This graph structure not only preserves the spatial distribution information of the elements but also encodes the organizational hierarchy between elements, laying a solid data structure foundation for subsequent feature learning and semantic inference based on graph neural networks. The node attributes and edge relationships in the element spatial relationship graph together constitute a structured abstract representation of the user interface layout, enabling machine learning models to understand the layout logic and element collaboration relationships of the interface from a global perspective.

[0037] Establish nested directed edges between parent and child nodes of visual element pairs with nested hierarchical relationships, including: For parent and child visual elements with a parent-child index relationship, a first nested directed edge pointing from the parent node to the child node and a second nested directed edge pointing from the child node to the parent node are respectively established in the element space relationship graph to form a parent-child bidirectional index structure. The first nested directed edge carries the relative position offset vector and relative size ratio vector of the child node in the coordinate system of the parent node, and the second nested directed edge carries the layout constraint type identifier and constraint reference direction applied by the parent node to the child node. Based on the bounding box coordinate parameters of the parent node visual element and the bounding box coordinate parameters of the child node visual element, the proportion of the area of ​​the child node bounding box contained inside the parent node bounding box to the total area of ​​the parent node bounding box is calculated, as well as the spacing offset of the four boundaries of the child node bounding box relative to the corresponding four boundaries of the parent node bounding box. The area proportion and the spacing offset of each boundary are encoded into the edge attribute vector of the first nested directed edge. The edge attribute vector is fused with the visual appearance feature vector of the corresponding node in the subsequent spatial context feature encoding process of the node to characterize the spatial constraint strength and boundary alignment relationship between the parent and child nodes.

[0038] During the parsing of UI design source files, visual elements commonly exhibit nested hierarchical relationships of containment and being contained. For example, container elements may contain child elements such as buttons and text boxes, or panel elements may contain multiple card components. To accurately represent this hierarchical spatial organization structure, a dedicated nesting relationship expression mechanism needs to be established in the element spatial relationship diagram.

[0039] When it is detected that the bounding box of one visual element completely contains the bounding box of another visual element, a parent-child nesting relationship is determined. At this point, in the element spatial relationship graph, a first nested directed edge is established with the parent node as the starting point and the child node as the ending point. This directed edge not only represents the hierarchical containment relationship but also carries attribute information describing the spatial distribution characteristics of the child node within the parent node. Specifically, with the top-left corner of the parent node's bounding box as the origin of the local coordinate system, the horizontal and vertical offset distances of the top-left corner of the child node's bounding box relative to this origin are calculated; these two distance values ​​constitute a relative position offset vector. Simultaneously, the ratio of the width of the child node's bounding box to the width of the parent node's bounding box and the ratio of the height of the child node's bounding box to the height of the parent node's bounding box are calculated; these two ratios constitute a relative size ratio vector. The relative position offset vector and the relative size ratio vector are encoded together into the attributes of the first nested directed edge, enabling this directed edge to quantitatively describe the specific placement of the child node within the parent node's coordinate space.

[0040] To achieve bidirectional traversal capability of the hierarchical structure, a second nested directed edge is established, pointing from child nodes to parent nodes, forming a bidirectional index structure between parent and child nodes. The attribute information of the second nested directed edge focuses on characterizing the layout control method applied by the parent node to the child node. By analyzing the layout management strategy of the parent node, the type of layout constraint it adopts is identified, such as linear arrangement constraint, grid alignment constraint, fixed anchor point constraint, or free float constraint. The identified layout constraint type is encoded as a type identifier and used as one of the attributes of the second nested directed edge. Furthermore, the reference direction when the parent node applies layout constraints is determined, such as horizontal linear constraints from left to right, vertical stacking constraints from top to bottom, or radial distribution constraints with the center of the parent node as the anchor point. The constraint reference direction is encoded as a direction vector or direction angle value and stored in the attributes of the second nested directed edge. This bidirectional edge structure enables the transmission of global layout rules from parent nodes to child nodes and the feedback of local space occupancy information from child nodes to parent nodes during subsequent graph neural network message passing.

[0041] To more precisely characterize the spatial constraint strength and boundary alignment relationship between parent and child nodes, a deep analysis of the geometric parameters of the node bounding boxes is required. Assume the coordinates of the top-left corner of the parent node's bounding box are... Width is The height is Then its lower right corner coordinates are The area of ​​the bounding box is The coordinates of the top-left corner of the child node's bounding box are... Width is The height is The coordinates of the lower right corner are The area of ​​the bounding box is Calculate the ratio of the bounding box area of ​​the child node to the bounding box area of ​​the parent node. This area ratio reflects the relative scale importance of the child node within the parent node's internal space; a larger ratio indicates that the child node is closer to filling the entire space of the parent node.

[0042] Next, we calculate the offsets of the four boundaries of the child node's bounding box relative to the corresponding boundaries of the parent node's bounding box. The horizontal spacing between the left boundary of the child node's bounding box and the left boundary of the parent node's bounding box is... The vertical distance between the upper boundary of the child node's bounding box and the upper boundary of the parent node's bounding box is... The horizontal spacing between the right boundary of the child node's bounding box and the right boundary of the parent node's bounding box is... The vertical distance between the bottom boundary of the child node's bounding box and the bottom boundary of the parent node's bounding box is... These four spacing offsets form a vector. This quantitatively describes the specific positioning state of the child node's bounding box within the parent node. When and When the values ​​are close, it indicates that the child node is approximately centered and aligned with the parent node horizontally; when... and When the values ​​are close, it indicates that the child node is approximately centered and aligned with the parent node in the vertical direction. If a certain spacing value is close to zero, it means that the corresponding boundary of the child node is close to the boundary of the parent node, presenting a boundary-aligned layout pattern.

[0043] area ratio offset from the four boundaries Combined into a five-dimensional vector This serves as the edge attribute vector of the first nested directed edge. This edge attribute vector not only carries the spatial occupancy information of the child node within the parent node but also implicitly contains the alignment intent and spacing rules in the layout design. During subsequent node feature update operations in the graph neural network, when encoding the spatial context features of a node, the edge attribute vectors of all adjacent edges of that node are aggregated. For a child node, the edge attribute vector of its connected first nested directed edge... It performs concatenation or weighted summation operations with the visual appearance feature vectors of the child nodes to achieve the fusion of edge attribute information and node features. This fusion mechanism enables the spatial context features of a node to simultaneously perceive its relative position, relative size, and alignment with the parent node's boundary within the parent node's coordinate system, thereby providing richer spatial constraint clues for subsequent semantic category inference.

[0044] The parent-child node bidirectional index structure plays multiple roles in practical applications. During top-down layout traversal, the first nested directed edge allows for rapid location of all child nodes of the parent node, obtaining the precise placement parameters of each child node within the parent coordinate system, facilitating the generation of code snippets conforming to the hierarchical structure. During bottom-up feature aggregation, the second nested directed edge can propagate local features of child nodes back to the parent node, enabling the semantic representation of the parent node to comprehensively consider the visual and spatial characteristics of all its child elements. For example, when a container node contains multiple button child nodes arranged horizontally with equal spacing, analyzing the edge attribute vectors of each child node... and The value indicates that the parent container uses a uniformly distributed horizontal linear layout constraint. This layout constraint information is fed back to the parent node through the second nested directed edge, enhancing the confidence that the parent node is identified as a "horizontal container" type during the semantic category inference stage.

[0045] The introduction of edge attribute vectors also improves the model's robustness to complex nested layouts. In some source design files, child nodes may not be strictly centered around their parent nodes, but may have specific margins or be deliberately offset to one side to achieve an asymmetrical visual effect. By accurately calculating and retaining the spacing offsets of the four boundaries, the model can learn these subtle layout preferences and accurately map the corresponding margin parameters to padding, margin, or alignment attribute values ​​when generating native layout code for the target platform, ensuring that the rendered interface maintains a high degree of visual consistency with the original source design file.

[0046] When dealing with complex UI structures with multiple nested layers, each pair of parent and child nodes is connected by an independent bidirectional edge, forming a multi-level tree-like topology. By recursively applying the above-mentioned edge attribute vector calculation and encoding process, the entire hierarchical spatial organization system from the top-level container to the bottom-level atomic components can be completely depicted in the element spatial relationship graph, laying a solid structured data foundation for subsequent joint representation learning based on graph neural networks.

[0047] The visual appearance features of each node in the element spatial relationship graph are encoded to generate a visual appearance feature vector. Based on the spatial context topology of each node in the element spatial relationship graph, a spatial context feature vector is generated, including: For each node, extract the following three types of visual appearance features from the corresponding visual elements: The geometric features of the bounding box include its aspect ratio, area, and tightness. Color distribution statistical characteristics, including color histogram distribution vector and dominant color identifier; Semantic features of text content, including the proportion of text regions and the number of text lines; The bounding box geometric features, the color distribution statistical features, and the text content semantic features are concatenated to generate the visual appearance feature vector of the node. Based on the element spatial relationship graph, for each node, a set of spatially adjacent nodes directly connected to the node through undirected edges and a set of nested hierarchical associated nodes directly connected to the node through nested directed edges are collected. The visual appearance feature vectors of each adjacent node in the spatially adjacent node set are aggregated by mean to generate a spatially adjacent aggregated feature vector. The visual appearance feature vectors of the parent node and child node in the nested hierarchical associated node set are aggregated by mean to generate a hierarchical associated aggregated feature vector. The spatially adjacent aggregated feature vector and the hierarchical associated aggregated feature vector are concatenated to generate the spatial context feature vector of the node.

[0048] After the element spatial relationship graph is constructed, deep feature extraction and encoding are required for each node in the graph. This process is divided into two parallel feature extraction paths: visual appearance feature encoding and spatial context topology structure encoding. The two are fused to form a complete node representation.

[0049] For visual appearance feature encoding, the bounding box geometry features are first extracted for the visual elements corresponding to each node. The aspect ratio of the bounding box is obtained by calculating the quotient of the width and height of the bounding box. This ratio reflects the morphological characteristics of the element. For example, the aspect ratio of a horizontally elongated element is usually greater than 3.0, while the aspect ratio of a square icon element is close to 1.0. The bounding box area is calculated by multiplying the width and height of the bounding box, in pixels squared, and is used to quantify the spatial scale occupied by the element on the canvas. Bounding box tightness measures the ratio of the actual pixel area occupied by the element to the area of ​​the bounding box. It is calculated by counting the number of non-transparent pixels within the bounding box and then dividing by the total area of ​​the bounding box. This indicator is used to distinguish between solid rectangular elements and wireframe outline elements. A tightness close to 1.0 is usually for filled buttons or color blocks, while a tightness below 0.3 may be for border-type input boxes or dividing lines.

[0050] Extracting statistical features of color distribution requires color space analysis of pixels within the element's bounding box. The RGB color space is quantized into a 16-level discretized color space, with each channel divided into four intervals, constructing a 64-dimensional color histogram distribution vector. By statistically analyzing and normalizing the pixel frequency of each color interval, a vector representation reflecting the element's color distribution characteristics is obtained. The dominant hue is determined by extracting the color interval with the highest frequency from the color histogram. This dominant hue is then converted to an HSV color space representation, where the hue value identifies the color family category, and saturation and lightness values ​​jointly describe the intensity of the color. For elements containing gradient fills, the dominant hue identifiers of the gradient's start and end colors are extracted, forming a duotone feature pair.

[0051] The extraction of semantic features from text content relies on the detection and analysis of text regions within the bounding boxes of elements. Text region detection algorithms are used to locate the boundaries of all text blocks within their bounding boxes. The sum of the areas of all text block bounding boxes is calculated and divided by the total area of ​​the element's bounding boxes to obtain the text region percentage. This percentage reflects the text density of the element; the text region percentage for tag-type elements typically exceeds 0.8, while that for icon-type elements is close to 0. The number of text lines is determined by detecting the number of newline characters within a text block or by a line segmentation algorithm based on the text baseline. Single-line text elements are marked as 1, and multi-line text paragraphs are labeled according to their actual line count. For purely graphic elements that do not contain text content, the text region percentage and the number of text lines are recorded as 0.

[0052] The three types of features are concatenated in a fixed order. The bounding box geometric features form a 3D vector, the color distribution statistical features include a 64-dimensional color histogram and a 3D dominant hue HSV, and the text content semantic features form a 2D vector. After concatenation, a 72-dimensional visual appearance feature vector is formed. This vector fully characterizes the visual appearance attributes of the node, providing a foundational feature for subsequent semantic category inference.

[0053] In the spatial context topology encoding stage, it is necessary to fully utilize the graph structure information of the element spatial relationship graph. For each node in the graph, its set of spatially adjacent nodes is first collected. All undirected edges connecting the node are traversed, and nodes directly connected by spatially adjacent edges are added to the set of spatially adjacent nodes. Spatial adjacency reflects the horizontal or vertical arrangement of elements in the visual layout; for example, multiple menu items in a navigation bar are spatially adjacent to each other, and labels and input boxes in a form also have spatial adjacency relationships. For isolated nodes or nodes connected only to parent and child nodes, their set of spatially adjacent nodes may be empty.

[0054] Simultaneously, collect the nested hierarchical related node set of the node. Traverse all directed edges connecting the node, identifying the incoming edges pointing to the node and the outgoing edges pointing from the node. The starting node of the incoming edge is the parent node of the node, reflecting which container element the node is contained in; the ending node of the outgoing edge is the set of child nodes of the node, reflecting which child elements are nested within the node. Combine the parent node and all child nodes to form the nested hierarchical related node set. For the root node, its parent node is empty; for the leaf node, its child node set is empty.

[0055] After obtaining the set of spatially adjacent nodes, the visual appearance feature vector of each adjacent node in the set is extracted, and a dimension-wise mean aggregation operation is performed on the feature vectors of all adjacent nodes. Specifically, the visual appearance feature vectors of all adjacent nodes are stacked in the first dimension to form a matrix, and the arithmetic mean is calculated for each column of the matrix to obtain an aggregated result vector with the same dimension as the original feature vectors, i.e., the spatially adjacent aggregated feature vector. This vector incorporates the visual feature information of surrounding elements, reflecting the local visual environment features of the current node. When the set of spatially adjacent nodes is empty, the spatially adjacent aggregated feature vector is filled with zero vectors.

[0056] For nested hierarchical node sets, parent node features and child node features need to be processed separately. If a node has a parent node, its visual appearance feature vector is directly extracted as the parent-level aggregated feature; if the node is the root node, the parent-level aggregated feature is filled with a zero vector. For the child node set, the visual appearance feature vectors of all child nodes are extracted, and the same dimensional mean aggregation operation is performed to obtain the child-level aggregated feature vector. When the child node set is empty, the child-level aggregated feature is filled with a zero vector. The parent-level aggregated feature vector and the child-level aggregated feature vector are concatenated sequentially to form the hierarchical aggregated feature vector. This vector encodes the hierarchical context information of the current node in the layout tree, including both the overall feature constraints of the container element and the statistical distribution features of its internal child elements.

[0057] Finally, the spatial adjacency aggregated feature vector and the hierarchical association aggregated feature vector are concatenated. Since the spatial adjacency aggregated feature vector has a dimension of 72, and the hierarchical association aggregated feature vector is formed by concatenating the 72-dimensional parent-level and 72-dimensional child-level aggregated features, totaling 144 dimensions, the concatenated spatial context feature vector has a total dimension of 216. This feature vector comprehensively captures the topological position information of nodes in the element spatial relationship graph, including horizontal and vertical adjacency relationships with surrounding elements, as well as nested containment relationships with parent containers and child elements.

[0058] In practical coding implementation, batch processing can be used to improve computational efficiency. The visual appearance feature vectors of all nodes in the element spatial relationship graph are organized into a node feature matrix, with the number of rows equal to the total number of nodes in the graph and 72 columns. Based on the graph's adjacency matrix and nesting relation matrix, the adjacent node index set and hierarchical association node index set corresponding to each node are pre-calculated. For spatial adjacency aggregation operations, feature sub-matrices of adjacent nodes are extracted in batches through matrix indexing and slicing operations, and the matrix mean function is called to aggregate along the node dimension, generating the spatial adjacency aggregation feature matrix for all nodes at once. For hierarchical association aggregation operations, the parent node index and child node index set are extracted separately, and batch aggregation is completed through matrix mapping and mean pooling. Finally, the spatial adjacency aggregation feature matrix and the hierarchical association aggregation feature matrix are merged along the feature dimension through tensor concatenation operations to obtain the spatial context feature matrix for all nodes.

[0059] Through the aforementioned feature encoding process, each node obtains a 72-dimensional visual appearance feature vector and a 216-dimensional spatial context feature vector. The visual appearance feature vector characterizes the node's intrinsic visual attributes, while the spatial context feature vector captures the node's extrinsic relational attributes. Both describe the node's semantic information from different perspectives, laying the representational foundation for subsequent feature fusion and semantic category inference. Visual appearance features provide low-level visual cues for identifying element types, while spatial context features provide high-level structural constraints for inferring element functions. The organic combination of the two can significantly improve the accuracy and robustness of semantic understanding.

[0060] The visual appearance feature vectors of each neighboring node in the spatial neighboring node set are aggregated by mean to generate a spatial neighboring aggregated feature vector, including: Based on the degree of adjacency between the bounding boxes of the node and each of the adjacent nodes in the spatial neighboring node set, the spatial proximity weight factor of each adjacent node is calculated. The spatial proximity weight factor is obtained by weighted summation of the proportion of the adjacency area between the bounding box of the node and the bounding boxes of the adjacent nodes and the proportion of the boundary adjacency length. The value of the spatial proximity weight factor is positively correlated with the degree of adjacency between the bounding boxes of the node and the adjacent nodes. Based on the adjacency degree of each adjacent node in the spatial relationship graph of the elements, the topological centrality weight factor of each adjacent node is calculated. The topological centrality weight factor is determined by the normalized ratio of the adjacency degree of each adjacent node to the sum of the adjacency degrees of all adjacent nodes in the spatial adjacent node set. The value of the topological centrality weight factor is positively correlated with the adjacency degree of the node. Each feature dimension component in the visual appearance feature vector of each adjacent node is multiplied by the corresponding spatial proximity weight factor and topological centrality weight factor. The weighted visual appearance feature vectors of each adjacent node are summed dimension by dimension. The result of the summation dimension by dimension is divided by the number of nodes in the spatial adjacent node set to obtain the spatial adjacent aggregated feature vector.

[0061] After obtaining the user interface design source files, the layout structure and visual element attributes are identified using parsing techniques, and a spatial adjacency mapping is established for each node in the element spatial relationship diagram. For any target node in the element spatial relationship diagram, its spatial adjacency relationship with surrounding nodes in the two-dimensional plane coordinate system is identified, determining the set of spatially adjacent nodes for that target node. The determination of the set of spatially adjacent nodes is based on the bounding box geometric position determination rules. Specifically, the degree of projection overlap between the bounding box of the target node and the bounding boxes of candidate adjacent nodes in the horizontal or vertical direction is calculated. When two bounding boxes have projection overlap in at least one direction and the minimum Euclidean distance between the bounding box boundaries is less than a preset distance threshold, they are determined to be spatially adjacent. The preset distance threshold is usually set to 1% to 3% of the canvas resolution, which can accommodate the element spacing commonly seen in visual design while avoiding misjudging elements that are too far apart in spatial position as adjacent nodes.

[0062] After determining the set of spatially adjacent nodes of the target node, the bounding box adjacency between the target node and each adjacent node in the set is calculated. Bounding box adjacency quantifies the closeness of two visual elements in geometric space, comprising two dimensions: adjacency area ratio and boundary adjacency length ratio. In calculating the adjacency area ratio, the overlapping projection area of ​​the target node's bounding box and the adjacent node's bounding boxes on the two-dimensional plane is first identified. When two bounding boxes have borders that are close but not overlapping, the overlapping projection area is recorded as zero. In calculating the boundary adjacency length ratio, the length of the common adjacency segment between the nearest boundaries of the two bounding boxes is measured. This segment is the length of the parallel or overlapping portion of the two bounding box boundaries when the distance is less than the boundary adjacency threshold, which is typically set to 2 to 5 pixels. The adjacency area ratio is obtained by dividing the overlapping projection area by the smaller of the areas of the two bounding boxes, with a value between 0 and 1. The boundary adjacency length ratio is obtained by dividing the common adjacency segment length by the smaller of the perimeters of the two bounding boxes, also normalized to the range of 0 to 1.

[0063] Based on the proportion of adjacent area and the proportion of boundary adjacent length, a weighted summation method is used to calculate the spatial proximity weight factor. Let the proportion of adjacent area be... The boundary adjacency length accounts for 1 / 3 of the total length. Spatial proximity weighting factor The calculation formula is: ; in and For the weighting coefficients, satisfying and The weighting coefficient is adjusted according to the characteristics of the layout type; for document-type layouts, it is usually set to... and Because visual elements in document-style layouts are mostly arranged linearly, the boundary adjacency length better reflects the relationship between elements; for card-style layouts, the boundary adjacency length is set... and This is because the area adjacency between card elements plays a dominant role in visual grouping. The value of the spatial proximity weight factor is positively correlated with the adjacency of the bounding boxes; when the bounding boxes of two visual elements are completely flush... The maximum value is achieved when two elements are spatially located within adjacent node sets but are actually far apart. The value is close to zero.

[0064] In addition to the spatial proximity weight factor, a topological centrality weight factor is introduced to characterize the importance of each neighboring node in the global topological structure of the element spatial relationship graph. The topological centrality weight factor is calculated based on the adjacency degree of each neighboring node. The adjacency degree is defined as the number of edges directly connected to that node in the element spatial relationship graph, reflecting the connection density of that node in establishing spatial relationships with other elements in the layout structure. For a given neighboring node in the set of spatially adjacent nodes, its adjacency degree is denoted as... The sum of the adjacency degrees of all adjacent nodes in the spatial adjacent node set is denoted as . The topological centrality weight factor of the adjacent node Calculated using the normalized ratio: ; in This represents the total number of neighboring nodes in the spatial neighbor set. This is for traversal indexing. The value of the topological centrality weight factor is positively correlated with the node's adjacency degree. Nodes with higher adjacency degrees occupy key connection positions in the layout structure, and their visual appearance features should be given higher weight during the aggregation process. Normalization ensures that the sum of the topological centrality weight factors of all adjacent nodes in the spatial adjacent node set is 1, which facilitates subsequent weighted aggregation calculations.

[0065] After obtaining the spatial proximity weight factor and topological centrality weight factor of each neighboring node, the visual appearance feature vectors of each neighboring node in the spatial neighboring node set are subjected to double weighting. The visual appearance feature vector is denoted as... , for one A dimensional vector, where This is the feature dimension, typically ranging from 128 to 512. The combined weighting factor is obtained by multiplying the spatial proximity weighting factor by the topological centrality weighting factor. : ; The comprehensive weighting factor considers both the geometrical proximity and the topological connectivity, ensuring that adjacent nodes that are both spatially close to the target node and centrally located in the topological structure receive the highest weight. For the ... The visual appearance feature vectors of each neighboring node are weighted, and the components of each feature dimension in the feature vector are weighted. Respectively with the comprehensive weighting factor Multiply to obtain the weighted eigenvector. : , ; All spatially adjacent nodes The weighted feature vectors of each neighboring node are summed along each feature dimension, and the weighted feature components of all neighboring nodes are accumulated along each feature dimension to obtain the aggregated feature vector. : , ; To eliminate the influence of differences in the number of neighboring nodes on the numerical scale of the aggregated feature vector, the result obtained by summing along each feature dimension is divided by the number of nodes in the spatial neighbor set. The mean aggregation operation is completed to generate the final spatially adjacent aggregated feature vector. : , ; Spatial adjacency aggregated feature vector The visual appearance feature distribution of spatially adjacent elements around a target node is encoded. Through a dual-weighting and mean aggregation mechanism, both spatial proximity information and topological centrality information are preserved, providing a visual semantic representation of the local neighborhood for subsequent spatial context feature encoding. In practical applications, when the set of spatially adjacent nodes of a target node is empty, the spatial proximity aggregation feature vector is initialized as a zero vector with the same dimension as the visual appearance feature vector, indicating that the node is isolated in the layout and lacks spatial adjacent context. For complex UI interfaces containing a large number of visual elements, an upper limit can be set on the number of adjacent nodes, selecting only those with the highest comprehensive weight factor value. Each adjacent node participates in the aggregation calculation. The value is usually set to 8 to 15 to retain the most relevant spatial neighborhood information while ensuring computational efficiency.

[0066] Semantic category inference is performed based on joint representation vectors, and element type labels are output. A structured layout tree carrying semantic information is constructed based on the element type labels and the element spatial relationship graph, including: Based on the joint representation vector of each visual element, the probability value of each visual element belonging to each preset semantic category is calculated, and the preset semantic category corresponding to the maximum probability value is determined as the element type label of the visual element. The preset semantic categories include container, text, image, list and interactive control categories. The node corresponding to the visual element with the smallest document object model hierarchy depth value in the element spatial relationship graph is taken as the root node of the structured layout tree. The parent-child hierarchy relationship between each node is determined according to the direction of the nested directed edges in the element spatial relationship graph. Each node is inserted into the corresponding hierarchy position in the structured layout tree layer by layer according to the parent-child hierarchy relationship. Each inserted node carries the element type label, spatial position parameters and style attribute set of the corresponding visual element in the structured layout tree. Perform hierarchical semantic consistency verification on the structured layout tree. If there is a predefined semantic conflict pattern between the element type label of the target node and the element type label combination of each child node in the set of child nodes of the target node, then re-extract the spatial context feature vector of the target node, and re-infer the semantic category of the target node and update its element type label based on the re-extracted spatial context feature vector.

[0067] After obtaining the joint representation vector of each visual element, a fully connected neural network layer maps the joint representation vector to the semantic category space. Specifically, a fully connected classification layer with five output units is set up, corresponding to five preset semantic categories: container, text, image, list, and interactive control. The joint representation vector is used as the input of the fully connected classification layer, and is added to the bias term through matrix multiplication to obtain a classification score vector with a dimension of 5. The Softmax activation function is applied to this classification score vector to convert the score values ​​of each dimension into classification probability values, and the sum of the classification probability values ​​is 1. The probability vector is traversed to find the classification probability value with the largest value and its corresponding index position. The preset semantic category corresponding to this index position is the element type label of the visual element. For example, if the five-dimensional classification probability vector of a visual element is [0.05, 0.72, 0.08, 0.10, 0.05], where the probability value of the second dimension is the largest (0.72) and this dimension corresponds to the text category, then the element type label of the visual element is determined to be the text category. For each node in the element spatial relationship graph, the semantic category inference process described above is performed, assigning a corresponding element type label to each node. After assigning element type labels, a structured layout tree is constructed based on the element spatial relationship graph. First, all nodes in the element spatial relationship graph are traversed, and the document object model (DOM) level depth value carried by each node is checked. This depth value records the nesting level position of the node in the DOM tree structure of the original design file. The node with the smallest depth value is selected as the root node of the structured layout tree, which usually corresponds to the top-level container element of the entire user interface. The root node is inserted at the root position of the structured layout tree, and the attribute information of the root node is initialized, including its element type label, spatial position parameters (including horizontal coordinate, vertical coordinate, width, and height), and style attribute set (including visual style information such as background color, border style, and margin parameters).

[0068] Next, based on the nested directed edges in the element spatial relationship graph, the parent-child hierarchical relationship between nodes is determined. Nested directed edges in the element spatial relationship graph point from parent nodes to child nodes, indicating that the visual element corresponding to the child node is spatially contained within the visual element corresponding to the parent node. All nested directed edges in the element spatial relationship graph are traversed, and for each nested directed edge, its starting node (parent node) and ending node (child node) are obtained. The position where the parent node has already been inserted is found in the structured layout tree, and the child node is inserted as a child of that parent node into the corresponding level position in the structured layout tree. When inserting a child node, the element type label, spatial position parameters, and style attribute set of the visual element corresponding to the child node are written into the node's attribute fields. Following a breadth-first traversal order, all nodes with the same depth value in the element spatial relationship graph are processed layer by layer, ensuring that the construction process of the structured layout tree strictly follows the nested hierarchical structure of the original design file.

[0069] After the structured layout tree is constructed, a hierarchical semantic consistency check is performed to detect and correct any potential semantic label errors. The specific check process is as follows: traverse all non-leaf nodes in the structured layout tree. For each non-leaf node, obtain the node's element type label and the element type labels of all child nodes in the node's child node set. Based on a predefined semantic conflict rule base, determine whether the combination of the node's element type label and its child node element type labels constitutes a semantic conflict pattern. Predefined semantic conflict patterns include: if the parent node's element type label is text, but its child node set contains child nodes with element type labels of container or list types, then a semantic conflict is determined; if the parent node's element type label is image, but its child node set is not empty, then a semantic conflict is determined; if the parent node's element type label is interactive control, but its child node set contains child nodes with element type labels of container types, then a semantic conflict is determined.

[0070] When a semantic conflict pattern is detected between a target node and its set of child nodes, the semantic category re-inference process for that target node is triggered. First, the spatial context feature vector of the target node is re-extracted from the element spatial relationship graph. Specifically, the set of spatially adjacent nodes of the target node in the element spatial relationship graph is re-collected. Based on the visual appearance feature vectors and spatial relationship weights of these spatially adjacent nodes, the spatial context feature vector of the target node is recalculated through a weighted aggregation operation. When recalculating the spatial context feature vector, the influence weight of the feature information of each child node in the target node's set of child nodes is increased, allowing the re-inference process to more fully consider the semantic information of the child nodes. The re-extracted spatial context feature vector is then re-fused with the original visual appearance feature vector of the target node to obtain an updated joint representation vector.

[0071] Based on the updated joint representation vector, the semantic category inference process is re-executed. The updated joint representation vector is input into the fully connected classification layer, and after matrix operations and the Softmax activation function, a new five-dimensional attribution probability vector is obtained. To avoid semantic conflicts after re-inference, a semantic constraint is introduced when selecting the maximum attribution probability value: the preset semantic category that forms a conflict pattern with the child node element type label is excluded from the candidate categories, and only the category with the highest attribution probability value among the remaining legal candidate categories is selected as the updated element type label. For example, if the target node's child node set contains multiple child nodes with the element type label "text", then during re-inference, the candidate categories of "text" and "image" are excluded, and only the category with the highest attribution probability is selected from the container, list, and interactive control categories. The updated element type label is written to the attribute field of the target node in the structured layout tree, completing the semantic label correction. For all nodes in the structured layout tree that are detected to have semantic conflicts, the above re-inference and label update operations are performed respectively until the entire structured layout tree passes the hierarchical semantic consistency check.

[0072] After completing the hierarchical semantic consistency check and correcting all semantic conflicts, the final structured layout tree is obtained. Each node in this structured layout tree carries an accurate element type label, complete spatial location parameters, and a detailed set of style attributes. The hierarchical structure of the tree strictly reflects the nesting hierarchy of visual elements in the original user interface design source file, and there are no semantic conflicts between the element type labels of all nodes and their child node sets. This structured layout tree provides a standardized intermediate representation for the subsequent generation of platform native layout code, ensuring that the generated layout code can accurately reproduce the semantic structure and visual appearance of the design file.

[0073] In the semantic category inference process, a confidence threshold mechanism can be introduced to address situations where boundary assignment probabilities are relatively close. Specifically, the difference between the highest and second-highest probability values ​​in the assignment probability vector is calculated. If this difference is less than a preset confidence threshold (e.g., 0.15), the semantic category of the visual element is considered ambiguous. For such ambiguous elements, the candidate semantic category list (including the top two categories by assignment probability) can be recorded in the node attributes during the construction of the structured layout tree for subsequent manual review or multimodal verification. This mechanism can effectively identify elements with high model prediction uncertainty and improve the robustness of the structured layout tree construction process.

[0074] Based on the rendering engine characteristics of the target rendering platform, the structured layout tree is mapped and converted into the native layout code of the target rendering platform, including: Obtain the list of native UI component types, component nesting constraint rule set, and style attribute naming conventions supported by the rendering engine of the target rendering platform. Based on the functional description of each native UI component in the list of native UI component types and the semantic description of each element type tag in the structured layout tree, establish a mapping relationship table from element type tags to native UI component types. A depth-first traversal is performed on the structured layout tree, and each node is processed in the order of traversal. The corresponding target native UI component type is obtained by querying the mapping table based on the element type label of the currently traversed node. The platform native UI component declaration code corresponding to the node is generated based on the target native UI component type. According to the parent-child hierarchy relationship of the currently traversed node in the structured layout tree, the platform native UI component declaration code corresponding to the child node is nested and written into the inner area of ​​the platform native UI component declaration code corresponding to the parent node. Each style attribute name in the style attribute set of the currently traversed node is converted into the native style attribute name used by the target rendering platform according to the style attribute naming specification. The converted native style attribute name and the corresponding style attribute value are then associated with the platform native UI component declaration code corresponding to the node in the form of key-value pairs.

[0075] After obtaining the structured layout tree carrying complete semantic information, it needs to be converted into native layout code that the target rendering platform can directly execute. Different rendering platforms have their own UI component systems, layout constraint mechanisms, and style attribute definition specifications, so the conversion process needs to be adapted and mapped according to the characteristics of the target platform.

[0076] First, obtain the rendering engine technical documentation of the target rendering platform and extract a complete list of all native UI component types supported by that platform. This list details the identifier name, functional description, supported attribute interfaces, and usage scenarios for each native UI component. For example, on a web platform, the list includes basic components such as div, span, button, and input; on an iOS platform, it includes native view classes such as UIView, UILabel, UIButton, and UITextField; and on an Android platform, it includes component classes such as View, TextView, Button, and EditText. Simultaneously, obtain the platform's component nesting constraint rule set. This rule set defines which component types can act as containers to contain child components, which component types must be leaf nodes, and the restrictions imposed on child component types by specific component types. For example, some platforms stipulate that text label components cannot nest other interactive components, or that the direct child nodes of list container components must be list item components. Furthermore, obtain the target platform's style attribute naming conventions document. This document specifies the standard naming conventions, value format requirements, and unit conversion rules for various style attributes on that platform.

[0077] After obtaining the complete platform technical specifications, a mapping table is established from element type tags in the structured layout tree to the native UI component types of the target platform. All element type tags appearing in the structured layout tree are traversed, and the functional semantic description of the node corresponding to each tag is extracted. For example, if a node's element type tag is "text display container," its semantic description is "a non-interactive area used to display static text content." This semantic description is then semantically matched with the functional descriptions of each component in the native UI component type list. For the web platform, "text display container" can be mapped to a span component or a p component; for the mobile platform, it can be mapped to a UILabel or a TextView component. When the semantic description of an element type tag matches the functional descriptions of multiple native components, priority is given based on the node's context and attribute characteristics in the layout tree. For example, if a node has multi-line text attributes and needs to support scrolling, a text container component with scrolling capabilities is prioritized. If a node contains clickable interactive attributes, a component type with click event response capabilities is selected. All mapping results are recorded in the mapping table, where each item includes the source element type tag, the target native component type, the mapping confidence score, and applicable condition constraints.

[0078] After constructing the mapping table, a depth-first traversal is performed on the structured layout tree. The traversal begins from the root node, pushing it onto the traversal stack. When the stack is not empty, the top node is popped as the current processing node. The element type label of the current node is obtained, and the corresponding target native UI component type is looked up in the mapping table. Based on the query result, the platform-specific native UI component declaration code for that node is generated. For web platforms, an HTML tag-style component declaration is generated, including start and end tags; for mobile platforms, object instantiation code in the corresponding programming language is generated, including constructor calls for the component class and necessary initialization parameter settings.

[0079] When generating component declaration code, the nesting relationship of the node in the layout tree needs to be handled. It checks if the current node has a parent node; if so, it retrieves the generated component declaration code corresponding to the parent node. The component declaration code of the current node is then written as a nested child component into the inner area of ​​the parent node's component declaration code. For declarative syntax platforms that support tag nesting, the child component tag is inserted between the start and end tags of the parent component tag; for platforms using imperative APIs, method call code is generated to add the child component object to the parent component container. During the nesting process, the component nesting constraint rule set is checked to verify whether the current combination of parent and child component types conforms to platform constraints. If a constraint violation exists, an adapter component is inserted between the parent and child nodes. This adapter component can function as both a valid child component of the original parent node and a valid parent container of the atomic node.

[0080] After processing component declarations and nested hierarchy, the style property set of the current node is transformed. This set stores all visual style settings for the node, including size and positioning properties, color appearance properties, and typography properties. Each style property item in the set is traversed, extracting its name and value. The property name is then matched against the specified names in the style property naming conventions document. For example, the property name "background color" used in the structured layout tree corresponds to "background-color" in the CSS specification for the web platform, "backgroundColor" on the iOS platform, and "setBackgroundColor" on the Android platform. The appropriate native property name is then selected and replaced based on the target platform type.

[0081] For attribute value conversion, it's necessary to handle unit and format differences between different platforms. Size attribute values ​​may be stored in device-independent pixels in the layout tree, requiring conversion based on the target platform's unit system. Web platforms may use multiple units such as px, em, and rem; mobile platforms may use density-independent units such as pt and dp. Color attribute values ​​may be represented as RGB three-channel values ​​in the layout tree; when converting to a web platform, they need to be formatted as hexadecimal color codes or RGB function calls; when converting to a mobile platform, a platform-native color object needs to be constructed. Font attributes may involve cross-platform mapping of font family names, mapping generic font family names to the target platform's built-in fonts or loaded custom font resource paths.

[0082] After converting attribute names and values, the conversion results are associated with the platform's native UI component declaration code as key-value pairs. For platforms that support inline styles, the style key-value pairs are directly written to the `style` attribute of the component tag; for platforms with separate style definitions, an independent style definition block is generated, a unique style identifier is assigned to each component, and the component is associated with the style definition through the identifier. In mobile platforms, the style attribute values ​​are assigned to the component instance item by item by calling the component object's property setting methods.

[0083] Continue the depth-first traversal, pushing all child nodes of the current node onto the traversal stack in the order defined in the layout tree. Repeat the above process of component declaration generation, nesting handling, and style attribute conversion until the traversal stack is empty, indicating that all nodes in the layout tree have been converted. Assemble all the generated platform-native UI component declaration code into a complete layout code file according to the traversal order. For the web platform, assemble it into an HTML document structure, including document type declaration, html root tag, head, and body; for the mobile platform, assemble it into the view constructor method of the view controller class or a layout description file.

[0084] During code assembly, add necessary platform-specific declarations and references. Web platforms require character encoding declarations, viewport meta tags, and links to external stylesheets and script files; mobile platforms require necessary framework import statements, class declarations, and lifecycle method definitions. Format the generated code, adding appropriate indentation, line breaks, and comments according to the target platform's coding style to ensure good readability and maintainability. Perform static code analysis to verify the syntax correctness of generated component declarations, the spelling accuracy of attribute names, and the legality of nested structures, automatically correcting or generating warnings for any potential issues detected. The final output native layout code can be directly provided to the target rendering platform's rendering engine for parsing and rendering, achieving accurate user interface presentation.

[0085] A second aspect of this invention provides an AI-based intelligent rendering system for UI layout information, comprising: The analytical modeling unit is used to obtain and parse the user interface design source file, extract layout structure information and visual element attribute information, and perform spatial relationship modeling on each visual element based on the layout structure information to generate an element spatial relationship diagram. In the element spatial relationship diagram, each visual element is represented by a node and the spatial adjacency relationship and nesting hierarchy relationship between visual elements are represented by an edge. The feature encoding unit is used to encode the visual appearance features of each node in the element spatial relationship graph to generate a visual appearance feature vector, and to encode the spatial context topology structure of each node in the element spatial relationship graph to generate a spatial context feature vector. The visual appearance feature vector and the spatial context feature vector are fused to obtain a joint representation vector. A semantic construction unit is used to perform semantic category inference based on joint representation vectors, output element type labels, and construct a structured layout tree carrying semantic information based on the element type labels and the element spatial relationship graph. Each node of the structured layout tree contains element type labels, spatial location parameters and style attribute sets. The code conversion unit is used to map the structured layout tree into the native layout code of the target rendering platform according to the rendering engine characteristics of the target rendering platform. The native layout code includes platform native UI component declarations and hierarchical nesting relationships between components that correspond one-to-one with each node in the structured layout tree.

[0086] A third aspect of the present invention provides an electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0087] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0088] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An AI-based intelligent rendering method for UI layout information, characterized in that, include: The user interface design source file is obtained and parsed to extract layout structure information and visual element attribute information. Based on the layout structure information, spatial relationship modeling is performed on each visual element to generate an element spatial relationship diagram. In the element spatial relationship diagram, nodes represent each visual element and edges represent the spatial adjacency and nesting hierarchy relationships between visual elements. The visual appearance features of each node in the element spatial relationship graph are encoded to generate a visual appearance feature vector. Based on the spatial context topology of each node in the element spatial relationship graph, a spatial context feature vector is generated. The visual appearance feature vector and the spatial context feature vector are fused to obtain a joint representation vector. Semantic category inference is performed based on joint representation vectors, and element type labels are output. Based on the element type labels and the element spatial relationship graph, a structured layout tree carrying semantic information is constructed. Each node of the structured layout tree contains element type labels, spatial location parameters and style attribute sets. Based on the rendering engine characteristics of the target rendering platform, the structured layout tree is mapped and converted into the native layout code of the target rendering platform. The native layout code includes platform native UI component declarations corresponding one-to-one with each node in the structured layout tree and the hierarchical nesting relationship between components.

2. The method according to claim 1, characterized in that, Based on the layout structure information, spatial relationship modeling is performed on each visual element to generate an element spatial relationship diagram, including: Extract the bounding box coordinate parameters and document object model (DOM) hierarchical path of each visual element from the layout structure information. Based on the bounding box coordinate parameters of each visual element, calculate the horizontal and vertical bounding box adjacency of each visual element pairwise. Mark visual element pairs that satisfy the horizontal adjacency condition and the vertical adjacency condition as spatial adjacency. According to the hierarchical index identifier of each visual element in the DOM hierarchical path, mark visual element pairs that are in the same DOM hierarchical path and whose hierarchical index identifiers satisfy the parent-child index relationship as nested hierarchical relationships. Traverse all visual element pairs and construct the element spatial relationship graph with each visual element as a node. Establish spatially adjacent undirected edges between nodes corresponding to visual element pairs with spatial adjacency, and establish nested directed edges from parent nodes to child nodes between nodes corresponding to visual element pairs with nested hierarchical relationships. The bounding box coordinate parameters, document object model level depth value, and node adjacency degree of each node in the element spatial relationship graph are associated with the corresponding visual element. The node adjacency degree is the sum of the number of spatially adjacent undirected edges and the number of nested directed edges directly connected to the node.

3. The method according to claim 2, characterized in that, Establish nested directed edges between parent and child nodes of visual element pairs with nested hierarchical relationships, including: For parent and child visual elements with a parent-child index relationship, a first nested directed edge pointing from the parent node to the child node and a second nested directed edge pointing from the child node to the parent node are respectively established in the element space relationship graph to form a parent-child bidirectional index structure. The first nested directed edge carries the relative position offset vector and relative size ratio vector of the child node in the coordinate system of the parent node, and the second nested directed edge carries the layout constraint type identifier and constraint reference direction applied by the parent node to the child node. Based on the bounding box coordinate parameters of the parent node visual element and the bounding box coordinate parameters of the child node visual element, the proportion of the area of ​​the child node bounding box contained inside the parent node bounding box to the total area of ​​the parent node bounding box is calculated, as well as the spacing offset of the four boundaries of the child node bounding box relative to the corresponding four boundaries of the parent node bounding box. The area proportion and the spacing offset of each boundary are encoded into the edge attribute vector of the first nested directed edge. The edge attribute vector is fused with the visual appearance feature vector of the corresponding node in the subsequent spatial context feature encoding process of the node to characterize the spatial constraint strength and boundary alignment relationship between the parent and child nodes.

4. The method according to claim 1, characterized in that, The visual appearance features of each node in the element spatial relationship graph are encoded to generate a visual appearance feature vector. Based on the spatial context topology of each node in the element spatial relationship graph, a spatial context feature vector is generated, including: For each node, extract the following three types of visual appearance features from the corresponding visual elements: bounding box geometric features, including the aspect ratio, area, and tightness of the bounding box; color distribution statistical features, including the color histogram distribution vector and the dominant color identifier; and text content semantic features, including the text area ratio and the number of text lines. Concatenate the bounding box geometric features, the color distribution statistical features, and the text content semantic features into feature vectors to generate the visual appearance feature vector of the node. Based on the element spatial relationship graph, for each node, a set of spatially adjacent nodes directly connected to the node through undirected edges and a set of nested hierarchical associated nodes directly connected to the node through nested directed edges are collected. The visual appearance feature vectors of each adjacent node in the spatially adjacent node set are aggregated by mean to generate a spatially adjacent aggregated feature vector. The visual appearance feature vectors of the parent node and child node in the nested hierarchical associated node set are aggregated by mean to generate a hierarchical associated aggregated feature vector. The spatially adjacent aggregated feature vector and the hierarchical associated aggregated feature vector are concatenated to generate the spatial context feature vector of the node.

5. The method according to claim 4, characterized in that, The visual appearance feature vectors of each neighboring node in the spatial neighboring node set are aggregated by mean to generate a spatial neighboring aggregated feature vector, including: Based on the degree of adjacency between the bounding boxes of the node and each of the adjacent nodes in the spatial neighboring node set, the spatial proximity weight factor of each adjacent node is calculated. The spatial proximity weight factor is obtained by weighted summation of the proportion of the adjacency area between the bounding box of the node and the bounding boxes of the adjacent nodes and the proportion of the boundary adjacency length. The value of the spatial proximity weight factor is positively correlated with the degree of adjacency between the bounding boxes of the node and the adjacent nodes. Based on the adjacency degree of each adjacent node in the spatial relationship graph of the elements, the topological centrality weight factor of each adjacent node is calculated. The topological centrality weight factor is determined by the normalized ratio of the adjacency degree of each adjacent node to the sum of the adjacency degrees of all adjacent nodes in the spatial adjacent node set. The value of the topological centrality weight factor is positively correlated with the adjacency degree of the node. Each feature dimension component in the visual appearance feature vector of each adjacent node is multiplied by the corresponding spatial proximity weight factor and topological centrality weight factor. The weighted visual appearance feature vectors of each adjacent node are summed dimension by dimension. The result of the summation dimension by dimension is divided by the number of nodes in the spatial adjacent node set to obtain the spatial adjacent aggregated feature vector.

6. The method according to claim 1, characterized in that, Semantic category inference is performed based on joint representation vectors, and element type labels are output. A structured layout tree carrying semantic information is constructed based on the element type labels and the element spatial relationship graph, including: Based on the joint representation vector of each visual element, the probability value of each visual element belonging to each preset semantic category is calculated, and the preset semantic category corresponding to the maximum probability value is determined as the element type label of the visual element. The preset semantic categories include container, text, image, list and interactive control categories. The node corresponding to the visual element with the smallest document object model hierarchy depth value in the element spatial relationship graph is taken as the root node of the structured layout tree. The parent-child hierarchy relationship between each node is determined according to the direction of the nested directed edges in the element spatial relationship graph. Each node is inserted into the corresponding hierarchy position in the structured layout tree layer by layer according to the parent-child hierarchy relationship. Each inserted node carries the element type label, spatial position parameters and style attribute set of the corresponding visual element in the structured layout tree. Perform hierarchical semantic consistency verification on the structured layout tree. If there is a predefined semantic conflict pattern between the element type label of the target node and the element type label combination of each child node in the set of child nodes of the target node, then re-extract the spatial context feature vector of the target node, and re-infer the semantic category of the target node and update its element type label based on the re-extracted spatial context feature vector.

7. The method according to claim 1, characterized in that, Based on the rendering engine characteristics of the target rendering platform, the structured layout tree is mapped and converted into the native layout code of the target rendering platform, including: Obtain the list of native UI component types, component nesting constraint rule set, and style attribute naming conventions supported by the rendering engine of the target rendering platform. Based on the functional description of each native UI component in the list of native UI component types and the semantic description of each element type tag in the structured layout tree, establish a mapping relationship table from element type tags to native UI component types. A depth-first traversal is performed on the structured layout tree, and each node is processed in the order of traversal. The corresponding target native UI component type is obtained by querying the mapping table based on the element type label of the currently traversed node. The platform native UI component declaration code corresponding to the node is generated based on the target native UI component type. According to the parent-child hierarchy relationship of the currently traversed node in the structured layout tree, the platform native UI component declaration code corresponding to the child node is nested and written into the inner area of ​​the platform native UI component declaration code corresponding to the parent node. Each style attribute name in the style attribute set of the currently traversed node is converted into the native style attribute name used by the target rendering platform according to the style attribute naming specification. The converted native style attribute name and the corresponding style attribute value are then associated with the platform native UI component declaration code corresponding to the node in the form of key-value pairs.

8. An AI-based intelligent rendering system for UI layout information, used to implement the method as described in any one of claims 1-7, characterized in that, include: The analytical modeling unit is used to obtain and parse the user interface design source file, extract layout structure information and visual element attribute information, and perform spatial relationship modeling on each visual element based on the layout structure information to generate an element spatial relationship diagram. In the element spatial relationship diagram, each visual element is represented by a node and the spatial adjacency relationship and nesting hierarchy relationship between visual elements are represented by an edge. The feature encoding unit is used to encode the visual appearance features of each node in the element spatial relationship graph to generate a visual appearance feature vector, and to encode the spatial context topology structure of each node in the element spatial relationship graph to generate a spatial context feature vector. The visual appearance feature vector and the spatial context feature vector are fused to obtain a joint representation vector. A semantic construction unit is used to perform semantic category inference based on joint representation vectors, output element type labels, and construct a structured layout tree carrying semantic information based on the element type labels and the element spatial relationship graph. Each node of the structured layout tree contains element type labels, spatial location parameters and style attribute sets. The code conversion unit is used to map the structured layout tree into the native layout code of the target rendering platform according to the rendering engine characteristics of the target rendering platform. The native layout code includes platform native UI component declarations and hierarchical nesting relationships between components that correspond one-to-one with each node in the structured layout tree.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images