A control block method and system of a UI interface

Abstract

The application provides a control block method and system of a UI interface, comprising: obtaining geometric information of each control to construct a control information sequence; calculating geometric characteristics of any two controls; based on a Gaussian distribution model, calculating similarity probability of the controls in each geometric characteristic according to the geometric characteristics, and fusing to obtain block probability between the two controls; taking each control in the control information sequence as a node, and taking the block probability between the two controls as the weight of the directed edge to construct a weighted directed graph; determining a classification threshold based on control block information of a historical UI interface, converting the weighted directed graph into a weighted undirected graph according to a preset selection mode and the classification threshold; analyzing the weighted undirected graph through a graph connected component algorithm, extracting each connected component as a block, and obtaining block information of the UI interface, so that the method realizes intelligent conversion from a UI design draft to structured block information, and improves page performance and response efficiency.

Description

Technical Field

[0001] This invention relates to the field of web page data processing technology, and in particular to a method and system for segmenting UI controls. Background Technology

[0002] In current artificial intelligence research, generating page code from UI design drafts usually relies on the correspondence between "controls and code". First, various controls on the interface are identified, and then the corresponding front-end structure and style are generated according to their type and position. With the help of machine learning methods similar to "image search", the geometric information such as the starting coordinates, width and height of each control are detected from the UI design interface.

[0003] Existing methods can identify the geometric information of a single control, but cannot determine whether multiple controls belong to the same functional module or visual area. They cannot differentiate different functional areas through targeted styles or layouts, resulting in a chaotic visual hierarchy and reduced aesthetics. Furthermore, they are difficult to implement dynamic loading by block, which affects page performance and response efficiency, and reduces the dynamism of the page and the overall interactive experience. Summary of the Invention

[0004] In view of this, the present invention proposes a method and system for control segmentation of UI interface. By identifying controls in the design drawing and extracting their geometric information, the probability of controls sharing the same block is calculated using a Gaussian distribution model. Then, a weighted directed graph is constructed and converted into an undirected graph. Finally, the segmentation structure is automatically extracted based on the graph connected component algorithm. This realizes the intelligent conversion from UI design draft to structured segmentation information, improves page performance and response efficiency, as well as page dynamism and overall interactive experience.

[0005] The technical solution of the present invention is implemented as follows: Firstly, a method for segmenting controls in a UI interface, comprising the following steps: S1, Based on the object detection network model, the UI interface is used to identify controls and obtain the geometric information of each control to construct a control information sequence; S2, based on the control information sequence, calculate the geometric features of any two controls in the sequence; S3, based on the Gaussian distribution model, calculates the similarity probability between controls on each geometric feature according to the geometric features, and then fuses them to obtain the merging probability between the two controls; S4. Using each control in the control information sequence as a node and the probability of merging between two controls as the weight of the directed edge, construct a weighted directed graph. S5: Determine the classification threshold based on the control block information of the historical UI interface, and transform the weighted directed graph into a weighted undirected graph according to the preset selection mode and classification threshold. S6 analyzes the weighted undirected graph using the graph connected component algorithm, extracts each connected component as a block, and obtains the block information of the UI interface.

[0006] Based on the above technical solutions, preferably, step S1, which involves identifying controls on the UI interface using a target detection network model to obtain the geometric information of each control in order to construct a control information sequence, includes the following sub-steps: S11, Input the UI image to be processed into the object detection network model; S12, the input UI interface image is detected using an object detection network model, each control is identified, and the geometric information of each control is output. The geometric information includes control type, position coordinates and size information. The position coordinates include the horizontal and vertical coordinates of the upper left corner of the control, and the size information includes the width and height of the control. S13 generates a unique identifier for all identified controls and constructs a sequence of control information using geometric information.

[0007] Based on the above technical solutions, preferably, step S2, which involves calculating the geometric features of any two controls in the control information sequence pair, includes: calculating the absolute value of the difference between the horizontal coordinates of the top left corners of the two controls to obtain the horizontal interval; calculating the absolute value of the difference between the vertical coordinates of the top left corners of the two controls to obtain the vertical interval; calculating the absolute value of the difference between the widths of the two controls to obtain the width difference; calculating the absolute value of the difference between the heights of the two controls to obtain the height difference; and constructing a set of geometric features based on the horizontal interval, vertical interval, width difference, and height difference between each control.

[0008] Based on the above technical solutions, preferably, step S3, which involves calculating the similarity probability between controls on each geometric feature based on the Gaussian distribution model and fusing them to obtain the merging probability between the two controls, includes the following sub-steps: S31, For each control in the control information sequence, calculate the geometric feature variance between it and other controls. The geometric feature variance includes the horizontal interval variance, the vertical interval variance, the width difference variance, and the height difference variance. S32, For any two controls in the control information sequence, calculate the corresponding horizontal interval similarity probability based on the corresponding horizontal interval and horizontal interval variance; S33, For any two controls in the control information sequence, calculate the corresponding vertical interval similarity probability based on the corresponding vertical interval and vertical interval variance; S34. For any two controls in the control information sequence, determine whether the width difference between the two controls is 0. If it is 0, the corresponding width difference similarity probability is 1; otherwise, calculate the corresponding width difference similarity probability based on the corresponding width difference and the width difference variance. S35, for any two controls in the control information sequence, determine whether the height difference between the two controls is 0. If it is 0, the corresponding height difference similarity probability is 1; otherwise, calculate the corresponding height difference similarity probability based on the corresponding height difference and the height difference variance. S36. Based on the product of the corresponding horizontal spacing similarity probability, vertical spacing similarity probability, width difference similarity probability, and height difference similarity probability, the block probability between each control is calculated respectively. Based on the above technical solutions, preferably, the step S5 of determining the classification threshold based on the control block information of the historical UI interface includes the following sub-steps: Based on the block information marked in the historical UI interface, extract all pairs of controls that are spatially adjacent and belong to the same block, calculate the block merging probability between each pair of controls, and obtain the first block merging probability set. Extract all pairs of controls that are spatially adjacent but belong to different blocks, calculate the merging probability between each pair of controls, and obtain the second merging probability set; Calculate the average of the minimum probability value in the first set of probability combinations and the maximum probability value in the second set of probability combinations, and use the average as the classification threshold.

[0009] Based on the above technical solutions, preferably, step S5, which involves converting a weighted directed graph into a weighted undirected graph according to a preset selection mode and classification threshold, includes: the preset selection mode includes a strict selection module and a wide selection mode; if the strict selection module is selected, an undirected edge is established when the probability of merging in both directions between two controls is higher than the classification threshold; if the wide selection mode is selected, an undirected edge is established when the probability of merging in either direction between two controls is higher than the classification threshold; the weight of the undirected edge is the smaller value between the probabilities of merging in both directions between the two controls; all nodes of the weighted directed graph are traversed to convert the weighted directed graph into a weighted undirected graph.

[0010] Based on the above technical solutions, preferably, step S6 involves analyzing the weighted undirected graph using a graph connected component algorithm, extracting each connected component as a block, and obtaining the block information of the UI interface, including the following sub-steps: S61, initialize the block result list and node access status, and mark all nodes in the weighted undirected graph as unvisited; S62, traverse all nodes in the weighted undirected graph, and construct the corresponding block for each unvisited node as the starting point of the search; S63: Add all search starting points to the queue, check if the queue is empty. If the queue is not empty, take out the first node of the queue and add it to the current block. Connect all neighbor nodes of the current node according to the undirected edges, mark the unvisited neighbor nodes as visited and add them to the queue, until the queue is empty, and the block construction is completed.

[0011] S64 adds the completed current block to the block result list until all nodes have been accessed, and the final output block result list serves as the block information for the UI interface.

[0012] Secondly, the present invention also provides a UI interface control segmentation system, implemented using a UI interface control segmentation method, comprising: The control recognition module is used to recognize controls on the UI interface based on the object detection network model, and to obtain the geometric information of each control to construct a control information sequence. The geometric feature calculation module is used to calculate the geometric features of any two controls in the control information sequence based on the control information sequence. The same block probability calculation module is used to calculate the similarity probability between controls on each geometric feature based on the Gaussian distribution model, and then fuse them to obtain the combined block probability between the two controls; The weighted directed graph construction module is used to construct a weighted directed graph by using each control in the control information sequence as a node and the probability of merging between two controls as the weight of the directed edge. The weighted undirected graph conversion module is used to determine the classification threshold based on the control block information of the historical UI interface, and convert the weighted directed graph into a weighted undirected graph according to the preset selection mode and classification threshold. The connected component extraction module is used to analyze the weighted undirected graph using the graph connected component algorithm, extract each connected component as a component, and obtain the component information for the UI interface.

[0013] Thirdly, the present invention also provides an electronic device, including at least one processor, at least one memory, a communication interface, and a bus; wherein the processor, memory, and communication interface communicate with each other through the bus; the memory stores a UI interface control segmentation method program that can be executed by the processor, and the UI interface control segmentation method program is configured to implement the UI interface control segmentation method.

[0014] Fourthly, the present invention also provides a computer-readable storage medium storing a UI interface control segmentation method program, which, when executed, implements the UI interface control segmentation method.

[0015] The UI control segmentation method and system of the present invention have the following advantages over the prior art: (1) By identifying the controls in the design drawing and extracting their geometric information, the probability of the same block between controls is calculated using the Gaussian distribution model, and then a weighted directed graph is constructed and converted into an undirected graph. Finally, the block structure is automatically extracted based on the graph connected component algorithm, realizing the intelligent conversion from UI design draft to structured block information, improving page performance and response efficiency, as well as the page dynamics and overall interactive experience. (2) Using the Gaussian distribution model, the geometric differences between controls are quantified into independent probability values, and then fused into a comprehensive probability of the same block by multiplying the probabilities. This method transforms discrete spatial relationships into continuous correlation strength measures, providing accurate weight basis for subsequent graph model construction, and enhancing the analysis ability and robustness of the block algorithm for complex layouts. (3) The adaptive classification threshold is determined by data-driven approach, and the weighted directed graph is converted into an undirected graph by adopting a strict / wide selection dual-mode strategy. The threshold is automatically learned using historical labeled data, replacing subjective manual setting. The connection strength is flexibly controlled through the two modes, thereby generating a graph structure that matches the downstream task, improving the adaptability and practicality of the block method to different UI styles and task requirements. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the steps of the UI control segmentation method of the present invention. Figure 2 A flowchart for generating the control set of the control segmentation method for the UI interface of the present invention; Figure 3 The flowchart of the directed graph for constructing the UI control segmentation method of the present invention is shown below. Figure 4 The flowchart of the undirected graph construction method for the UI control segmentation method of the present invention is shown below. Figure 5 This is a flowchart illustrating the process of extracting block information for the UI control block segmentation method of the present invention. Detailed Implementation

[0018] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0019] like Figure 1-5 As shown, in the first aspect, a method for segmenting controls in a UI interface includes the following steps: S1, based on the object detection network model, performs control recognition on the UI interface, obtains the geometric information of each control to construct a control information sequence.

[0020] Step S1 includes the following sub-steps: S11, Input the UI image to be processed into the object detection network model; It should be noted that the object detection network model is a lightweight YOLO11-based model based on deep learning. This model includes a standard feature extraction backbone network, a neck network for multi-scale feature fusion, and a detection head module that integrates bounding box prediction, classification prediction, and distribution focus loss calculation. The model weights are initialized using weights pre-trained on a large general dataset to leverage its learned general feature representation capabilities and improve convergence speed and performance on UI control detection tasks. The dataset structure used for model training and inference is defined by the data.yaml configuration file, which specifies the image paths, annotation file paths, and control category lists. Before being input into the network, all UI interface images are uniformly scaled to a fixed size of 960×960 pixels. During training, data is loaded in batches of 8 using a single thread. To improve the model's generalization ability to different UI design styles, various data augmentation strategies are used during the training phase, including HSV color space adjustment and random horizontal flipping.

[0021] The total number of training epochs was set to 120, with an early termination tolerance of 30 epochs. This means that if the validation set performance does not improve within 30 consecutive epochs, training will be terminated early to prevent overfitting. The optimizer used was AdamW, with an initial learning rate of 0.001 and a weight decay coefficient of 0.0005. The training process employed a learning rate warm-up strategy, and automatic mixed precision (AMP) training was used where possible to accelerate computation and save GPU memory. In addition, a fixed random seed was used to ensure the reproducibility of the experimental process and results.

[0022] The model is trained using a multi-task joint loss function, which consists of a weighted sum of three sub-losses: bounding box regression loss, classification loss, and distribution focus loss. The bounding box regression loss has a weight of 7.5 and is used to accurately predict the position and size of the control. The classification loss has a weight of 0.5 and is used to correctly identify the type of control. The distribution focus loss has a weight of 1.5 and is used to optimize the probability distribution representation of the bounding box position to improve positioning accuracy. These losses work together to guide the update of model parameters through the backpropagation algorithm.

[0023] This model is pre-trained using a large amount of labeled UI image data and can recognize various preset types of UI controls, including but not limited to buttons, text boxes, images, labels, drop-down menus, and checkboxes. Users upload UI images to be processed through a graphical interface or API interface. The system automatically adjusts the images to the fixed size required by the model and performs normalization preprocessing before inputting them into the object detection network model.

[0024] S12, the input UI interface image is detected using an object detection network model, each control is identified, and the geometric information of each control is output. The geometric information includes control type, position coordinates and size information. The position coordinates include the horizontal and vertical coordinates of the upper left corner of the control, and the size information includes the width and height of the control. It should be noted that the object detection network model performs forward inference on the input UI interface image and outputs the bounding box information and category confidence of each detected control. Specifically, for each identified control, the model outputs the control type, position coordinates, and size information. The position coordinates include the horizontal and vertical coordinates of the upper left corner of the control, and the size information includes the width and height of the control. Low-confidence detection results are filtered out according to a preset confidence threshold to ensure recognition accuracy.

[0025] S13 generates a unique identifier for all identified controls and constructs a sequence of control information using geometric information.

[0026] It should be noted that the system iterates through all identified controls, assigns a globally unique identifier to each control, and then organizes all control identifiers and their complete geometric information into an ordered list, i.e., a control information sequence, according to the recognition order. The data structure of this sequence can be in the form of a JSON array.

[0027] like Figure 2 As shown, the specific process of this embodiment is as follows: 1.1 Organize existing controls, clarify the UI presentation, control type, and control code of each control, and train the YOLO11 model based on known samples so that it can recognize the corresponding controls in the UI design page; the control code information will be used for code generation after the subsequent block determination. 1.2 After the UI designer completes the interface design, the generated UI design draft is submitted to this system as input; 1.3. Using the trained YOLO model, the input UI design is inspected; the model can automatically identify the top-left corner coordinates, width, height, and control type of each control in the design. 1.4 After completing the control recognition, all recognized controls are numbered sequentially starting from 1 until all controls are numbered. Each control element has a unique number, control type, top-left pixel coordinates, width, and height, which can be used as input data for subsequent geometric feature analysis and block modeling.

[0028] In this embodiment, by employing a trained YOLO11 lightweight object detection network, the complete geometric information of diverse controls can be efficiently and accurately identified and extracted from UI design drawings, laying a high-quality data foundation for subsequent analysis.

[0029] S2, calculate the geometric features of any two controls in the sequence based on the control information sequence.

[0030] Step S2, which involves calculating the geometric features of any two controls in the control information sequence, includes: calculating the geometric features of any two controls in the control information sequence. The horizontal spacing is calculated by calculating the absolute value of the difference between the top-left corner x-coordinates of the two controls. The expression is: DX i,j =| x i - x j |; In the formula, x i For the first i The horizontal coordinate of the top left corner of each control x j For the first j The horizontal coordinate of the top left corner of each control; The vertical interval is calculated by calculating the absolute value of the difference between the top-left corner ordinates of the two controls. The expression is as follows: DY i,j =|y i -y j |; In the formula, y iFor the first i The top-left y-coordinate of the control j No. j The top-left vertical coordinate of each control; Calculate the absolute value of the width difference between the two controls to obtain the width difference expression; DW i,j =|w i -w j |; In the formula, w i For the first i Width of each control, y j No. j Width of each control; Calculate the absolute value of the height difference between the two controls to obtain the height difference expression; DH i,j =|h i -h j |; In the formula, h i For the first i The height of each control, y j No. j Each control's height; A set of geometric features is constructed based on the horizontal spacing, vertical spacing, width difference, and height difference between each control.

[0031] S3, based on the Gaussian distribution model, calculates the similarity probability between controls on each geometric feature according to the geometric features, and then merges them to obtain the merging probability between the two controls.

[0032] Step S3 includes the following sub-steps: S31, For each control in the control information sequence, calculate the geometric feature variance between it and other controls. The geometric feature variance includes the horizontal interval variance, the vertical interval variance, the width difference variance, and the height difference variance. S32, for any two controls in the control information sequence, calculate the corresponding horizontal interval similarity probability based on the corresponding horizontal interval and horizontal interval variance, expressed as: ; In the formula, For other controls in the control i The variance of the horizontal interval.

[0033] S33, for any two controls in the control information sequence, calculate the corresponding vertical interval similarity probability based on the corresponding vertical interval and vertical interval variance, expressed as: ; In the formula, For other controls in the control i The variance of the longitudinal interval.

[0034] S34, for any two controls in the control information sequence, determine whether the width difference between the two controls is 0. If it is 0, the corresponding width difference similarity probability is 1; otherwise, calculate the corresponding width difference similarity probability based on the corresponding width difference and the variance of the width difference, as expressed by: ; In the formula, For controls i The variance relative to the width differences of all other controls; S35, for any two controls in the control information sequence, determine whether the height difference between the two controls is 0. If it is 0, the corresponding height difference similarity probability is 1; otherwise, calculate the corresponding height difference similarity probability based on the corresponding height difference and the height difference variance, as expressed by: ; In the formula, For controls i The variance of the height relative to the height of all other controls.

[0035] S36. Based on the product of the corresponding horizontal spacing similarity probability, vertical spacing similarity probability, width difference similarity probability, and height difference similarity probability, the merging probability between each control is calculated, and the expression is as follows: .

[0036] It should be noted that, based on the geometric feature set, the variance of each control with all other controls in four dimensions—horizontal spacing, vertical spacing, width difference, and height difference—is first calculated as the scaling parameter of the Gaussian distribution. For any two controls, these variances are used to calculate their independent similarity probability in each of the four dimensions, with the width and height differences assigned the maximum probability value of 1. Assuming that the features are independent, the probability values ​​in the four dimensions are multiplied to obtain the combined probability, which comprehensively reflects the overall similarity between the two controls in terms of spatial position and size layout. This step transforms discrete geometric relationships into continuous probability measures through probabilistic modeling, effectively quantifying the intrinsic correlation strength between controls. This provides a precise basis for edge weights in the subsequent construction of a weighted directed graph, thereby improving the adaptability and accuracy of block division judgment for complex UI layouts.

[0037] S4. Using each control in the control information sequence as a node and the probability of merging between two controls as the weight of the directed edge, construct a weighted directed graph. It's important to note that within the same block, controls may not be directly adjacent, but rather indirectly linked through other controls. For example, suppose a block contains four controls: control 1 is adjacent to both control 2 and control 3, but not directly adjacent to control 4; control 2 and control 3 are both directly adjacent to control 4. In this case, although control 1 and control 4 are not directly adjacent, they are indirectly linked through control 2 and control 3, and should still be classified as belonging to the same block. Therefore, simply relying on the direct adjacency of controls cannot accurately identify blocks; a graph-based approach is used to describe the relationships between controls, including both direct adjacency and indirect linkage.

[0038] Based on controls i arrive j The probability of merging between P(T) i ,T j Construct a weighted directed graph G=(V,E); Nodes: Each control Ti in the control set is a node; Edges: Edges pointing from control Ti to control Tj, with weights equal to the merging probability P(Tj). i ,T j ); By traversing all control pairs (T) i ,T j (i≠j), calculate the corresponding block probability P(T) i ,T j Based on the calculation results, the probability of merging between control pairs is used as the weight of the edges to construct a weighted directed graph G=(V,E), where V is the set of controls; E is the set of directed edges between control pairs, and the weight of each edge is its merging probability P(T). i ,T j ).

[0039] like Figure 3 As shown, the specific process is as follows: 2.1 From the control set obtained in the previous stage, select controls in sequence as the current base control. If it is the first time processing, select the control with the number 1. Then, select the next record of the previous control each time. The information of the current base control includes: control number, top left corner coordinates, width and height. 2.2. Taking the current baseline control as the reference object, calculate the horizontal spacing, vertical spacing, width difference, and height difference between it and each other control in the set, and record the corresponding target control number to form a geometric feature set about the current baseline control. Each record includes: target control number, horizontal spacing, vertical spacing, width difference, and height difference. 2.3 Based on the obtained set of geometric features, statistics are performed on the four types of features: horizontal spacing, vertical spacing, width difference, and height difference, and their corresponding variances are calculated. These variances will be used as parameters when calculating the probability of the same block based on the Gaussian model. 2.4 From the geometric feature set, select one record at a time as the current target element. If it is the first time processing, select the first record in the set. Then, select the next record after the previous record each time. The record contains: the target control number, and the horizontal spacing, vertical spacing, width difference and height difference between the target control and the current reference control. 2.5. Using the geometric feature values ​​of the current baseline control and the current target control, as well as various variances, calculate the probability of being in the same block corresponding to the horizontal spacing, vertical spacing, width difference, and height difference respectively. Under the assumption that each geometric feature is independent of each other, multiply these four probabilities to obtain the overall probability of being in the same block between the current baseline control and the current target control. 2.6. Using the index of the current baseline control as the starting point of the directed edge and the index of the current target control as the ending point of the directed edge, and using the overall probability of the same block as the edge weight, construct a directed edge from the baseline control to the target control. 2.7 Check whether all records in the geometric feature set corresponding to the current baseline control have been processed. If not, return to step 2.4 to select the next geometric feature record and construct the corresponding directed edge; if all records have been processed, proceed to the next step.

[0040] 2.8 Check if all controls in the entire control set have been used as the base control for the above calculation and connection; if there are still controls that have not been processed as the base control, return to step 2.1, select the next control as the new base control, and repeat steps 2.2 to 2.7; if all controls have been processed, proceed to step 2.9.

[0041] 2.9. All directed edges generated for each baseline control and its target control in the above process are summarized. Using the control as the node and the probability of being in the same block as the edge weight, a complete weighted directed graph is finally obtained, which provides the basic structure for subsequent undirected graph transformation and block extraction.

[0042] In this embodiment, based on the sequence of control information, four geometric features—horizontal spacing, vertical spacing, width difference, and height difference—are calculated between any two controls to construct a complete set of geometric features. Subsequently, a Gaussian distribution model is introduced, and the variance of the geometric features of each control relative to other controls is calculated as a distribution parameter to evaluate the independent similarity probability of two controls in each feature dimension, and these are merged into a combined probability that comprehensively reflects the strength of their spatial and dimensional association. Finally, using controls as nodes and combined probabilities as directed edge weights, a complete weighted directed graph is constructed by traversing all control pairs. By using probabilistic modeling to transform discrete geometric relationships into a continuous graph structure, not only are the direct adjacency relationships between controls quantified, but indirect associations are also captured, thus providing an accurate and robust data foundation for subsequent graph analysis and effectively improving the understanding and adaptability of the block partitioning algorithm to complex and nested UI layouts.

[0043] S5: Determine the classification threshold based on the control block information of the historical UI interface, and transform the weighted directed graph into a weighted undirected graph according to the preset selection mode and classification threshold. Step S5 determines the classification threshold based on the control block information of the historical UI interface, including: Based on the block information marked in the historical UI interface, extract all pairs of controls that are spatially adjacent and belong to the same block, calculate the block merging probability between each pair of controls, and obtain the first block merging probability set. Extract all pairs of controls that are spatially adjacent but belong to different blocks, calculate the merging probability between each pair of controls, and obtain the second merging probability set; Calculate the average of the minimum probability value in the first set of probability combinations and the maximum probability value in the second set of probability combinations, and use the average as the classification threshold.

[0044] It should be noted that by learning the boundaries of probabilistic representation between the same block and different blocks from historical experience, the classification criteria are made more objective and adaptive. This avoids the blindness of manually setting fixed thresholds based on experience, enabling the block-segmentation algorithm to adapt to different design specifications and improving the generalization ability and practicality of the method.

[0045] Step S5, which involves converting the weighted directed graph into a weighted undirected graph based on a preset selection mode and classification threshold, includes: The preset selection modes include a strict selection module and a wide selection mode. If the strict selection module is selected, an undirected edge is established when the probability of merging in both directions between the two controls is higher than the classification threshold. If the wide selection mode is selected, an undirected edge is established when the probability of merging in either direction between the two controls is higher than the classification threshold. The weight of the undirected edge is the smaller value between the probabilities of merging in both directions between the two controls. All nodes of the weighted directed graph are traversed to transform the weighted directed graph into a weighted undirected graph.

[0046] It's important to note that strict mode is suitable for scenarios with high requirements for block segmentation accuracy, such as when a reliable structure is needed for subsequent layout inference, code generation, or interaction logic reasoning. In this case, the system uses a higher threshold and performs strict screening based on the consistency of bidirectional edge weights, only grouping controls into the same block when the probability of being in the same block is high enough and the bidirectional relationship is stable, in order to minimize erroneous merging. Wide mode, on the other hand, is suitable for scenarios with higher recall requirements, such as early coarse segmentation, auxiliary annotation, or tasks that need to avoid block breaks as much as possible. In this mode, the threshold is relatively low, the judgment conditions are more lenient, and it tends to merge related controls together first, and then refine or correct them in subsequent steps. The choice of mode can be determined based on specific task requirements, UI layout complexity, and the cost of misjudgment on the validation set. Strict mode is preferred when the cost of erroneous merging is higher, while wide mode is preferred when erroneous splitting is more unacceptable.

[0047] like Figure 4 As shown, the specific process is as follows: 3.1 In the known UI interface data, manually label two types of adjacent control pairs: adjacent control pairs that belong to the same block; and adjacent control pairs that do not belong to the same block. For the above two types of control pairs, apply the block merging probability calculation method mentioned above to obtain the minimum block merging probability among adjacent control pairs in the same block and the maximum block merging probability among adjacent control pairs in different blocks. Use the average of these two probabilities as the threshold W. 3.2, Enumerate each pair of candidate controls sequentially from the control collection ( i , j If it's the first selection, pair the first control with the second control; then, select the next pair of controls for each subsequent selection until all pairwise combinations have been traversed, recording the node number of the current control pair. i and j ; 3.3, In the constructed directed graph, find the node. i and nodes j Let the minimum value of the weights of the two directed edges between them be denoted as the node. i With nodes j The weight of the undirected edge between them; 3.4 Determine whether the undirected edge weight obtained in 3.3 is greater than the threshold W. If so, then consider the node... i With nodes j If nodes are adjacent and belong to the same block, proceed to step 3.5; otherwise, consider them as nodes. i With nodes j They do not constitute an adjacent relationship in the sense of being in the same block; 3.5, for control pairs that meet the conditions, by node i and nodes jUsing the two endpoints and the obtained undirected edge weights as edge weights, construct undirected edges and add them to the undirected edge set; 3.6 Check if all control pairs have been processed. If there are a total of n If there are n nodes, the number of node pairs is (n*(n-1)). If all node pairs have been traversed, proceed to step 3.7. If there are still unprocessed node pairs, return to step 3.2 and continue to select the next pair of nodes. 3.7. All undirected edges are summarized, and a complete weighted undirected graph is constructed by using controls as nodes and the filtered undirected edges as connection relationships and edge weights.

[0048] In this embodiment, based on historical labeled data, the minimum merging probability of adjacent control pairs within the same block and the maximum merging probability of adjacent control pairs between different blocks are statistically analyzed, and the average of the two is calculated as an adaptive classification threshold, thereby achieving objective data-driven discrimination criteria. According to the preset strict or broad selection mode, all node pairs in the weighted directed graph are traversed, and the minimum bidirectional merging probability is compared with the threshold. Based on the logic of different modes, it is decided whether to establish undirected edges between nodes, and the minimum probability value is used as the weight of the undirected edge. Finally, a weighted undirected graph is constructed. This step eliminates the arbitrariness of manually setting the threshold through adaptive threshold and balances the segmentation accuracy and recall rate by using a dual-mode conversion strategy. This allows the graph structure to intelligently adapt to different UI design styles and downstream task requirements, providing a clean and task-aligned input for the final highly robust connected component segmentation, and improving the practicality, generalization ability and operability of the entire segmentation method.

[0049] S6 analyzes the weighted undirected graph using the graph connected component algorithm, extracts each connected component as a block, and obtains the block information of the UI interface.

[0050] Step S6 includes the following sub-steps: S61, initialize the block result list and node access status, and mark all nodes in the weighted undirected graph as unvisited; S62, traverse all nodes in the weighted undirected graph, and construct the corresponding block for each unvisited node as the starting point of the search; S63: Add all search starting points to the queue, check if the queue is empty. If the queue is not empty, take out the first node of the queue and add it to the current block. Connect all neighbor nodes of the current node according to the undirected edges, mark the unvisited neighbor nodes as visited and add them to the queue, until the queue is empty, and the block construction is completed.

[0051] S64 adds the completed current block to the block result list until all nodes have been accessed, and the final output block result list serves as the block information for the UI interface.

[0052] like Figure 5 As shown, the specific process is as follows: 4.1 Create a dictionary `Visited` of length `n`, where the key is the control number and the value is a boolean value to record whether the corresponding control node has been visited in the undirected graph. `Visited[i] = true` indicates that the control has been visited and assigned to a block; `Visited[i] = false` indicates that the control has not yet been assigned to a block. During initialization, set the `Visited` value of all controls to `false`, indicating that none of them are assigned to a block. 4.2 Create an empty list `Blocks` to store the final block results. Each element in `Blocks` represents a block; each block element is itself a list of control indices, which may contain only one control or multiple controls. 4.3 Select a control from the control set in sequence and mark its index as i. If it is the first selection, set i=1. Then select the next control after the current control in each subsequent selection until all controls have been selected in sequence. 4.4 If Visitid[i]=false, it means that the control has not yet been segmented, and a connected component (segmentation) search needs to be performed starting from this control, and proceed to step 4.5; if Visitid[i]=true, it means that the control has been assigned to a certain segment, and no repeated processing is needed, and proceed directly to step 4.16.

[0053] 4.5 Initialize two data structures for the current block: CurrentB stores the sequence numbers of all controls in the current block; DTmp serves as a queue to record the control nodes to be traversed, used for breadth-first search / depth-first search in the undirected graph: Initialize CurrentB as an empty list; initialize the current control... i Add it to the queue DTmp as the starting point for traversal; 4.6 Mark the current starting point control i as visited and set Visited[i]=true, indicating that the node has entered the search process of the current block; 4.7 Check if there are still nodes to be processed in the queue DTmp. If DTmp is empty, it means that the current control and all its connected adjacent controls have been traversed and all controls in the current block have been found. Proceed to step 4.8. If DTmp is not empty, it means that there are still connected nodes to be processed. Proceed to step 4.9 and continue to expand the current block. 4.8. Add the current block CurrentB as a complete block to the Blocks list. After the operation is completed, end the construction process of the current block and proceed to step 4.16 to check if there are any unprocessed controls.

[0054] 4.9 Pop a record from the queue DTmp and record the control's serial number as X; 4.10, Add node X to CurrentB, indicating that control X belongs to the same block as other controls in the current block; 4.11. In the undirected graph, traverse all nodes adjacent to node X. For each traversal, select an unprocessed adjacent node and record its index as Y. If it is the first traversal, select the first adjacent node of X. Then select the next adjacent node of X in turn until all adjacent nodes of X have been checked. After selecting the current adjacent node Y, proceed to step 4.12.

[0055] 4.12 Check if the adjacent node Y has been visited. If Visited[Y]=false, it means that the adjacent node has not been divided into blocks and needs to be included in the traversal of the current block. Proceed to step 4.13. If Visited[Y]=true, it means that the node has been processed and there is no need to repeat the traversal. Proceed directly to step 4.15. 4.13. Add the adjacent node Y to the queue DTmp so that Y and its adjacent nodes can be traversed further in the future; 4.14 Mark node Y as visited and set Visited[Y]=true, indicating that Y has been included in the search scope of the current block; 4.15. Determine whether all nodes adjacent to node X have been traversed. If all adjacent nodes of X have been checked, return to step 4.7 and continue to check whether queue DTmp is empty. If there are still untraversed adjacent nodes, return to step 4.11 and continue to select the next adjacent node Y for processing.

[0056] 4.16 After completing the construction of a block, check whether all controls have been block-based: if all controls have Visited[i] as true, it means that all controls have been assigned to a block, and proceed to step 4.17; if there are still controls with Visited[i] = false, it means that there are still controls that have not been block-based, return to step 4.3, select the next unprocessed control as the new starting point, and continue to execute the block-based process.

[0057] 4.17. All block results stored in the list Blocks are used as the final output. Each element corresponds to a block; the set of controls in each block is the set of controls that are interconnected in the undirected graph; the single-element block corresponds to the isolated point control in the undirected graph. At this point, the undirected graph has been completely divided into several blocks, and all block results are stored in Blocks.

[0058] In this embodiment, the access state dictionary and the block result list are initialized. Then, all unvisited control nodes are traversed, and a breadth-first / depth-first search is started from each node. During the search, a queue is used to dynamically maintain the nodes to be explored. The unvisited neighbors of the current node are continuously included in the traversal range and their states are marked, thereby gradually aggregating all controls in the connected area to form a block. When the queue is empty, the current block is completed and stored in the result list. This process is repeated until all nodes are visited. This process efficiently and reliably transforms the topological connection relationship of the weighted undirected graph into a control set partition with clear semantics. It not only ensures that the same control is correctly aggregated, but also preserves the independent blocks of isolated controls. The final output structured block information provides a directly usable foundation for the component-based encapsulation, layout generation, and interaction logic binding of the front-end code, improving the automation level and output quality of the conversion from design draft to code.

[0059] Secondly, the present invention also provides a UI interface control segmentation system, implemented using a UI interface control segmentation method, comprising: The control recognition module is used to recognize controls on the UI interface based on the object detection network model, and to obtain the geometric information of each control to construct a control information sequence. The geometric feature calculation module is used to calculate the geometric features of any two controls in the control information sequence based on the control information sequence. The same block probability calculation module is used to calculate the similarity probability between controls on each geometric feature based on the Gaussian distribution model, and then fuse them to obtain the combined block probability between the two controls; The weighted directed graph construction module is used to construct a weighted directed graph by using each control in the control information sequence as a node and the probability of merging between two controls as the weight of the directed edge. The weighted undirected graph conversion module is used to determine the classification threshold based on the control block information of the historical UI interface, and convert the weighted directed graph into a weighted undirected graph according to the preset selection mode and classification threshold. The connected component extraction module is used to analyze the weighted undirected graph using the graph connected component algorithm, extract each connected component as a component, and obtain the component information for the UI interface.

[0060] Thirdly, the present invention also provides an electronic device, including at least one processor, at least one memory, a communication interface, and a bus; wherein the processor, memory, and communication interface communicate with each other through the bus; the memory stores a UI interface control segmentation method program that can be executed by the processor, and the UI interface control segmentation method program is configured to implement the UI interface control segmentation method.

[0061] Fourthly, the present invention also provides a computer-readable storage medium storing a UI interface control segmentation method program, which, when executed, implements the UI interface control segmentation method.

[0062] It should be noted that this system corresponds to the aforementioned UI interface control segmentation method. All implementation methods in the above method embodiments are applicable to the embodiments of this system and can achieve the same technical effect.

[0063] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0064] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system and modules described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0065] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0066] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0067] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0068] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0069] Furthermore, it should be noted that in the system and method of the present invention, it is obvious that the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered equivalent solutions of the present invention. Moreover, the steps performing the above series of processes can naturally be executed in the order described, but are not necessarily required to be executed in chronological order; some steps can be executed in parallel or independently of each other. Those skilled in the art will understand that all or any step or component of the method and apparatus of the present invention can be implemented in any computing device (including processors, storage media, etc.) or network of computing devices, in hardware, firmware, software, or a combination thereof. This is something that those skilled in the art can achieve by using their basic programming skills after reading the description of the present invention.

[0070] Therefore, the object of the present invention can also be achieved by running a program or a set of programs on any computing system. The computing system can be a known general-purpose system. Therefore, the object of the present invention can also be achieved simply by providing a program product containing program code implementing the method or apparatus. That is, such a program product also constitutes the present invention, and the storage medium storing such a program product also constitutes the present invention. Obviously, the storage medium can be any known storage medium or any storage medium developed in the future. It should also be noted that in the apparatus and method of the present invention, it is obvious that the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered equivalent to the present invention. Furthermore, the steps performing the above series of processes can naturally be performed in the order described, but are not necessarily required to be performed in chronological order. Some steps can be performed in parallel or independently of each other.

[0071] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for segmenting controls in a UI interface, characterized in that, Includes the following steps: S1, Based on the object detection network model, the UI interface is used to identify controls and obtain the geometric information of each control to construct a control information sequence; S2, based on the control information sequence, calculate the geometric features of any two controls in the sequence; S3, based on the Gaussian distribution model, calculates the similarity probability between controls on each geometric feature according to the geometric features, and then fuses them to obtain the merging probability between the two controls; S4. Using each control in the control information sequence as a node and the probability of merging between two controls as the weight of the directed edge, construct a weighted directed graph. S5: Determine the classification threshold based on the control block information of the historical UI interface, and transform the weighted directed graph into a weighted undirected graph according to the preset selection mode and classification threshold. S6 analyzes the weighted undirected graph using the graph connected component algorithm, extracts each connected component as a block, and obtains the block information of the UI interface.

2. The UI interface control segmentation method as described in claim 1, characterized in that, Step S1, which involves identifying UI controls based on an object detection network model and obtaining the geometric information of each control to construct a control information sequence, includes the following sub-steps: S11, Input the UI image to be processed into the object detection network model; S12, the input UI interface image is detected using an object detection network model, each control is identified, and the geometric information of each control is output. The geometric information includes control type, position coordinates and size information. The position coordinates include the horizontal and vertical coordinates of the upper left corner of the control, and the size information includes the width and height of the control. S13 generates a unique identifier for all identified controls and constructs a sequence of control information using geometric information.

3. The UI interface control segmentation method as described in claim 2, characterized in that, Step S2, which involves calculating the geometric features of any two controls in the control information sequence pair, includes: calculating the absolute value of the difference between the horizontal coordinates of the top left corners of the two controls to obtain the horizontal interval; calculating the absolute value of the difference between the vertical coordinates of the top left corners of the two controls to obtain the vertical interval; calculating the absolute value of the difference between the widths of the two controls to obtain the width difference; calculating the absolute value of the difference between the heights of the two controls to obtain the height difference; and constructing a set of geometric features based on the horizontal interval, vertical interval, width difference, and height difference between each control.

4. The UI interface control segmentation method as described in claim 3, characterized in that, Step S3, which involves calculating the similarity probability between controls on each geometric feature based on the Gaussian distribution model and fusing them to obtain the merging probability between two controls, includes the following sub-steps: S31, For each control in the control information sequence, calculate the geometric feature variance between it and other controls. The geometric feature variance includes the horizontal interval variance, the vertical interval variance, the width difference variance, and the height difference variance. S32, For any two controls in the control information sequence, calculate the corresponding horizontal interval similarity probability based on the corresponding horizontal interval and horizontal interval variance; S33, For any two controls in the control information sequence, calculate the corresponding vertical interval similarity probability based on the corresponding vertical interval and vertical interval variance; S34. For any two controls in the control information sequence, determine whether the width difference between the two controls is 0. If it is 0, the corresponding width difference similarity probability is 1; otherwise, calculate the corresponding width difference similarity probability based on the corresponding width difference and the width difference variance. S35, for any two controls in the control information sequence, determine whether the height difference between the two controls is 0. If it is 0, the corresponding height difference similarity probability is 1; otherwise, calculate the corresponding height difference similarity probability based on the corresponding height difference and the height difference variance. S36. Based on the product of the corresponding horizontal spacing similarity probability, vertical spacing similarity probability, width difference similarity probability, and height difference similarity probability, the block probability between each control is calculated respectively. The UI interface control segmentation method as described in claim 4 is characterized in that, the step S5 of determining the classification threshold based on the historical UI interface control segmentation information includes the following sub-steps: Based on the block information marked in the historical UI interface, extract all pairs of controls that are spatially adjacent and belong to the same block, calculate the block merging probability between each pair of controls, and obtain the first block merging probability set. Extract all pairs of controls that are spatially adjacent but belong to different blocks, calculate the merging probability between each pair of controls, and obtain the second merging probability set; Calculate the average of the minimum probability value in the first set of probability combinations and the maximum probability value in the second set of probability combinations, and use the average as the classification threshold.

5. The UI interface control segmentation method as described in claim 5, characterized in that, Step S5, which involves converting a weighted directed graph into a weighted undirected graph based on a preset selection mode and classification threshold, includes: the preset selection mode includes a strict selection module and a wide selection mode; if the strict selection module is selected, an undirected edge is established when the probability of merging in both directions between two controls is higher than the classification threshold; if the wide selection mode is selected, an undirected edge is established when the probability of merging in either direction between two controls is higher than the classification threshold; the weight of the undirected edge is the smaller of the probabilities of merging in both directions between the two controls; and all nodes of the weighted directed graph are traversed to convert the weighted directed graph into a weighted undirected graph.

6. The UI interface control segmentation method as described in claim 6, characterized in that, Step S6, which involves analyzing the weighted undirected graph using a graph connected component algorithm to extract each connected component as a block and obtain the block information for the UI interface, includes the following sub-steps: S61, initialize the block result list and node access status, and mark all nodes in the weighted undirected graph as unvisited; S62, traverse all nodes in the weighted undirected graph, and construct the corresponding block for each unvisited node as the starting point of the search; S63: Add all search starting points to the queue, check if the queue is empty. If the queue is not empty, take out the first node of the queue and add it to the current block. Connect all neighbor nodes of the current node according to the undirected edges, mark the unvisited neighbor nodes as visited and add them to the queue, until the queue is empty, and the block construction is completed. 7.S64, add the completed current block to the block result list until all nodes have been accessed, and finally output the block result list as the block information of the UI interface.

8. A UI control segmentation system, implemented using the UI control segmentation method as described in any one of claims 1-7, characterized in that, include: The control recognition module is used to recognize controls on the UI interface based on the object detection network model, and to obtain the geometric information of each control to construct a control information sequence. The geometric feature calculation module is used to calculate the geometric features of any two controls in the control information sequence based on the control information sequence. The same block probability calculation module is used to calculate the similarity probability between controls on each geometric feature based on the Gaussian distribution model, and then fuse them to obtain the combined block probability between the two controls; The weighted directed graph construction module is used to construct a weighted directed graph by using each control in the control information sequence as a node and the probability of merging between two controls as the weight of the directed edge. The weighted undirected graph conversion module is used to determine the classification threshold based on the control block information of the historical UI interface, and convert the weighted directed graph into a weighted undirected graph according to the preset selection mode and classification threshold. The connected component extraction module is used to analyze the weighted undirected graph using the graph connected component algorithm, extract each connected component as a component, and obtain the component information of the UI interface.

9. An electronic device, characterized in that, It includes at least one processor, at least one memory, a communication interface, and a bus; wherein the processor, memory, and communication interface communicate with each other through the bus; the memory stores a UI interface control segmentation method program that can be executed by the processor, and the UI interface control segmentation method program is configured to implement the UI interface control segmentation method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a UI interface control segmentation method program, which, when executed, implements the UI interface control segmentation method as described in any one of claims 1-7.

Images