UI element intelligent classification method and system based on semi-supervised learning and product

Abstract

The invention discloses a semi-supervised learning-based UI (user interface) element intelligent classification method, system and product, which are characterized in that on the basis of a Transform structure, a Mean Teacher consistency regularization mechanism is introduced to improve the generalization ability of a model under a small sample condition, and accurate classification and structured grouping of fragmented and overlapped UI elements are realized through multi-modal feature fusion. The method comprises the following steps: firstly, extracting multi-source features such as images, texts, colors, positions and categories from each UI element, and uniformly encoding the multi-source features into 256-dimensional embedded representation; inputting the embedded sequence into a multi-layer Transform encoder for context modeling, outputting three types of structure tags, and automatically finishing element grouping according to the three types of structure tags; according to the method, the accuracy and the stability of UI element classification can be effectively improved, the dependence on manual annotation is reduced, and the method is suitable for scenes such as automatic interface generation, low-code development, UI prototype analysis and interactive behavior modeling.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, machine learning and human-computer interaction interface, and relates to a user interface (UI) element automatic classification method, system and product, in particular to a method, system and product for automatically classifying and grouping user interface (UI) elements based on a semi-supervised learning method, which is suitable for various application scenarios such as automatic code generation, interface optimization, intelligent testing, etc. BACKGROUND

[0002] A user interface (UI) design prototype, as a visual blueprint of the front-end interaction part of a software system, is widely used in the early stages of the software development process. It plays an irreplaceable important role in communicating product design intentions, verifying user interaction processes, and shortening development cycles. With the rise of technologies such as "low code" and "automated development", more and more research and industrial practices have begun to try to directly convert UI design prototypes into front-end code, thereby realizing the automatic transition from interface design to interface implementation and significantly improving research and development efficiency. However, there are still several key problems in this automated process, the most prominent of which is the accurate recognition and classification of fragmented and overlapping UI elements in the prototype.

[0003] In actual UI design, especially when using mainstream design tools such as Sketch and Figma, a visual functional component is often composed of multiple fine-grained layers, for example, a button may be composed of multiple independent elements such as a border, text, and icons. This "fragmentation" phenomenon, although beneficial to visual presentation, has brought serious challenges to subsequent code generation. Traditional image recognition methods often cannot understand the semantic association between these fragments, resulting in chaotic code structure, incomplete component functionality, and reduced interface maintainability. At the same time, the "overlap" phenomenon between UI elements is also very common, and different elements may appear visually intersected or overlaid due to layout or decoration purposes, causing element boundary ambiguity and interfering with the classification model judgment. These problems not only affect the consistency of the interface's interaction logic and appearance, but also may cause technical risks such as layout errors and performance degradation at runtime.

[0004] To overcome the above problems, in recent years researchers have carried out a lot of work in the recognition of graphical user interface, UI component detection and classification. Traditional methods are mostly based on image processing and artificial rules, such as edge detection, color distribution, element alignment degree and other measurement indexes for element combination. However, these heuristic methods are difficult to cope with the modern UI design with various styles and complex layout. With the development of deep learning, convolutional neural network (CNN) and Transformer architecture are widely introduced to improve the accuracy of element detection and classification. For example, Owl Eyes, Nighthawk and other systems automatically detect UI display problems based on CNN model; target detection models based on Faster R-CNN and YOLOv5-MGC are also used for identification and classification of small components. These methods have improved the accuracy and automation of UI element recognition to some extent.

[0005] However, the current end-to-end UI element classification method based on deep learning still has two key bottlenecks: first, they are highly dependent on large-scale high-quality labeled data, and the complex features such as semantic relationship and combination structure of UI elements are difficult to efficiently label through crowdsourcing, resulting in insufficient label consistency and structural integrity in public data sets (such as RICO, Screen Recognition). Second, the existing methods mostly use strong supervision learning strategy, and once the labeled data is insufficient, the model performance will be greatly reduced, and the generalization ability is limited, which is difficult to adapt to the scene demand of UI style diversity and data scarcity in real projects.

[0006] In summary, the current UI element classification task still cannot provide stable and universal solutions when facing the complex challenges of fragmentation, overlapping and data scarcity. How to make full use of the potential structural information of unlabeled data under the condition of limited labeled data to improve the accuracy and generalization ability of UI element classification model has become a technical problem to be solved. SUMMARY

[0007] The present application aims to solve the technical problems of difficult identification of fragmented and overlapping elements, lack of labeled data and low accuracy of automatic classification in existing UI design prototypes. A UI element intelligent classification method, system and product based on semi-supervised learning are provided.

[0008] The technical scheme adopted by the method of the present application is: a UI element intelligent classification method based on semi-supervised learning, comprising the following steps: Step 1: obtaining UI elements, converting the multi-modal attributes of each UI element into embedded vectors in a unified format as input of a deep learning model; Step 2: inputting the embedded vectors in a unified format into the deep learning model to output the UI element classification result; The deep learning model consists of a position encoder, a multi-layer Transformer encoder and a SoftMax layer; the multi-layer Transformer encoder is a stacked model consisting of N Transformer encoder blocks of the same structure, each Transformer encoder block contains a multi-head self-attention mechanism layer MHA, a first normalization layer LN, a feedforward neural network FFN and a second normalization layer LN; wherein the output of the position encoder is connected to the output residual of the first normalization layer LN and then input into the feedforward neural network FFN; the input of the feedforward neural network FFN is connected to the output residual of the second normalization layer LN and then input into the SoftMax layer to output the classification result; N is a preset value; Step 3: Based on the classification results, group the output.

[0009] Preferably, the specific implementation of step 1 includes the following sub-steps: Step 1.1: Use a convolutional neural network model to extract the features of the image part of each UI element and obtain the image embedding vector ; Step 1.2: Convert the text content of each UI element into a dense vector suitable for convolutional neural network processing to obtain the text embedding vector ; Step 1.3: Extract the background color and bounding box coordinates of the element to obtain the color embedding vector and the bounding frame embedding vector ; Step 1.4: Input the element category label into the embedding layer to generate dimensional category embedding vector ; Step 1.5: Embed the five features of image, text, color, bounding box and category into equal length dimensional vectors, and performs element-wise weighted fusion on each vector to form a unified element-level embedding representation.

[0010] As a preference, in step 1.1, the image of each UI element is individually cropped and encoded using a precise alignment method; then the pre-trained ResNet-50 is used as the image encoding basic network, and the output dimension of the last fully connected layer of the ResNet-50 is Set to 256 to generate a fixed-dimensional visual embedding vector; all UI element images are uniformly scaled to a fixed size and processed by ResNet-50 to output a two-dimensional tensor , each row corresponds to the image code of a UI element; Indicates the number of UI elements, represents the embedding dimension of each element.

[0011] As preferred, in step 1.2, first, the text is standardized, including converting all to lowercase and removing special characters; then the pre-trained RoBERTa tokenizer is used for tokenization, and it is mapped to a 32-dimensional index vector representation; further, padding and truncation strategies are used to standardize all texts to fixed-length tensors; to realize the dimension alignment of text features and image features, finally, the compression strategy is used to sum each column of the tensor, compress the first dimension of the tensor, and finally output a two-dimensional tensor .

[0012] As preferred, in step 1.3, the color information is represented in channel, and each channel is normalized to the interval; if an element lacks color information, it is automatically filled with zero as its default vector; the bounding box information contains the absolute position and size of the element in the canvas, which is represented in the format , and is scaled to the standard range by the min-max normalization method, where represent the length and width, respectively; the color embedding vector and the bounding box embedding vector are processed by a fully connected layer, so that their output shapes are ; where represents the number of UI elements, represents the embedding dimension of each element.

[0013] As preferred, the specific implementation of step 2 includes the following sub-steps: Step 2.1: Use the position encoding strategy based on different frequency trigonometric functions to add the position encoding vector to the element features to generate the final embedding sequence containing position information; ; where represent the image embedding vector, color embedding vector, category embedding vector, bounding box embedding vector, and text embedding vector, respectively; Step 2.2: input the embedding sequence into the deep learning model to generate the final classification representation ; For the th block in the sequence, its output representation is , and represents the hidden state of this sub-layer; in the initial stage, the element embedding is sent to the first Transformer encoder block; after processing by N consecutive encoder blocks, the final hidden state ; Finally, the features are further mapped and transformed by a double-layer feedforward network FFN to generate the final classification representation ; ; ; ; denotes the hidden state of the k-th position in the final N-th layer output feedforward mapping to generate the representation for classification ; Step 2.3: Based on the function, the model output is mapped into a probability distribution ; each row of the probability distribution corresponds to the classification probability of an element, reflecting the probability of the element belonging to each of the three categories; wherein the output of the row is denoted as , where represents the classification label, which describes the predicted probability distribution of the element.

[0014] As preferred, in step 3, the output after grouping based on the label-driven grouping strategy; the grouping process starts from the element marked as "start merge", and sequentially merges the subsequent elements marked as "merge" into the current group until the next "start merge" label is encountered, at which time a new grouping unit is started; in this process, all elements marked as "non-merge" will be excluded from any grouping and kept separately.

[0015] As preferred, the deep learning model is a trained model; The training process includes the following steps: Step A: Constructing a UI prototype dataset; Step A1: UI prototype data acquisition and format parsing; By batch acquiring Sketch format UI design prototype files from the internet open source platform OurSketch website; by using SketchTool tool to decompress and parse the prototype files one by one, the element information of each page is extracted, including position, size, color, content, category attribute; Step A2: Element image and attribute information extraction; A depth-first traversal algorithm based on tree structure is adopted to expand the layer hierarchy level by level, from the top page to each nested layer; each traversed element exports its corresponding image snapshot, stacking structure, and JSON data containing detailed attributes; the detailed attributes include but are not limited to: element coordinates, size, layer order, background color, text content, and font style; each element data extracted is composed of three files: one is the prototype file, which shows the overall interface; the second is the element stacking diagram file, which shows the layer snapshot of a single element; the third is the element attribute file, which stores structured description information; Step A3: After completing the preliminary analysis, strict screening rules are adopted to ensure data quality, mainly including integrity check, outlier filtering, logical conflict processing, and invalid element elimination. Integrity check requires eliminating samples missing key files or attributes; outlier filtering ensures that the bounding box coordinates are normalized to the [0, 1] interval and the color channel values are valid; logical conflict processing manually reviews samples with visual overlap exceeding 50% or conflicting layer labels; invalid element elimination filters decorative layers with excessively small size or high transparency; Step A4: A manual fine-labeling method is used to construct a label set, and each element is assigned one of the three labels: "initial merge", "merge", and "non-merge"; for each group of elements that should be merged, the first appearing element is labeled as "initial merge", and the remaining members are labeled as "merge"; if an element does not belong to any merge group, it is labeled as "non-merge"; all elements are initially "non-merge", and the labeling personnel modify them according to semantic and visual logic; Step A5: Label quality review and data quality control; After completing the preliminary labeling, manual review is performed, and the reviewer reviews and evaluates the labels of the prototype according to the preset rules; if conflicts, inconsistencies, or omissions are found during the review process, they are immediately fed back to the labeling personnel for revision; Step B: Convert the multi-modal attributes of each UI element into a unified format embedding vector as input for the deep learning model, and use the Mean Teacher framework's semi-supervised training method to train the deep learning model; The semi-supervised training method of the Mean Teacher framework includes two models that share parameters but have different update strategies: the student model (Student) and the teacher model (Teacher); both models use the same UI element classification structure in the initial stage and are initialized through parameter copying; during training, the student model parameters are updated through standard backpropagation and gradient descent; the teacher model parameters are not directly trained through backpropagation, but are updated through the exponential moving average (EMA) method of the student model parameters, keeping the teacher model as a stable target for the student model's prediction; During training, the supervised loss Consistency loss Weighted fusion, construct the total training loss function Loss: ; Wherein, is the weight of the consistency loss, and the following formula is used to realize the dynamic adjustment mechanism with the growth of the training progress: ; ; Wherein, is the maximum value of the consistency weight, is the number of periods of gradual increase of the consistency weight, used to control the growth speed of the consistency weight in the early stage of training, so as to ensure that the consistency weight grows slowly in the early stage of training, and gradually accelerates the growth with the progress of training until reaching the maximum value, represents the number of current training iterations, function as the activation function to realize smooth increase.

[0016] For labeled samples , introduce data enhancement noise , get the perturbed sample , and input it into the student model for prediction to generate the output ; Then calculate the cross entropy loss between the prediction result and the true label : ; Wherein, is a classification loss function used to measure the difference between the model prediction probability distribution and the true label, and the model parameters are optimized by minimizing the negative log likelihood.

[0017] For unlabeled data , inject different noise to generate perturbed samples ; The sample is input into the student model and the teacher model respectively to produce prediction results and ; The consistency loss is obtained by calculating the mean square error MSE of the output results of the two: ; Wherein, represents the number of samples of unlabeled data in the current batch. In the training process, the student model updates its parameters ( represents the trainable parameters in the student model) after each iteration; At the same time, the parameters of the teacher model are updated smoothly according to the following formula: ; wherein, wherein denotes the parameters of the model in the i-th iteration, denotes denotes the parameters of the model in the i-th iteration, denotes denotes the EMA decay coefficient, which determines the “inertia” of the model parameters, used to control the weighted average of new and old parameters; the value of a gradually increases with the training round , and the specific update is: .

[0018] The technical scheme adopted by the system of the application is: a UI element intelligent classification system based on semi-supervised learning, comprising: one or more processors; a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the UI element intelligent classification method based on semi-supervised learning.

[0019] The technical scheme adopted by the product of the application is: a UI element intelligent classification product based on semi-supervised learning, comprising computer program instructions, which, when running on a computer, cause the computer to execute the UI element intelligent classification method based on semi-supervised learning.

[0020] Compared with the prior art, the beneficial effects of the application include: (1) The application adopts an accurate alignment method, encodes each element image after individual cropping, and effectively avoids interference information.

[0021] (2) The extraction process of the image features of the application first accurately cuts the corresponding area in the prototype image, scales it to a uniform size, and then inputs it into the ResNet-50 model, ensuring that the visual features are not disturbed by the background.

[0022] (3) The bounding box information of the application is recorded in the format of element top-left corner coordinates and width and height, and is input into the embedding layer after being mapped to the standard value range through minimum-maximum normalization, improving the sensitivity of the model to spatial relationships.

[0023] (4) Through the multi-modal fusion mechanism, the model can realize robust modeling of fragmented and overlapped UI elements while maintaining the integrity of the structure, providing a guarantee for improving the classification accuracy.

[0024] (5) The application adopts a label-driven automatic grouping strategy to ensure accurate aggregation of fragment components, and has good universality and scalability.

[0025] (6) In the training process, the format analysis is realized through the SketchTool tool, which is a command line interface tool provided by Sketch. The SketchTool can export the Sketch document to ZIP format and automatically parse it into multiple JSON structure files, thereby realizing fast access to layer structure, attribute data and image resources. Through the tool, structured information can be directly extracted from the Sketch file and the visual software interface can be skipped, thereby greatly improving the batch processing efficiency.

[0026] (7) In the training process, each element data extracted is composed of three files, and the trinity data format ensures comprehensive modeling of visual and semantic information in the classification task.

[0027] (8) In the training process, the label set is constructed by manual fine labeling, which not only ensures the consistency and reproducibility of labeling, but also provides clear boundaries for grouping and classification.

[0028] (9) In the training process, through double artificial processing, the label data has high accuracy, high consistency and good generalization, which provides a solid data foundation for subsequent semi-supervised training and model evaluation.

[0029] (10) In the training process, the dynamic adjustment mechanism ensures that the teacher model changes rapidly in the early training to quickly respond to the progress of the student model, and remains stable in the later training to provide reliable learning goals for the student model.

[0030] (11) In the training process, the consistency loss is adopted to encourage the student model to keep consistent with the teacher model in the face of input disturbance, thereby enhancing the robustness of the model.

[0031] (12) In the training process, the weight adjustment strategy is adopted to control the slow enhancement of the consistency signal in the early training, and gradually increase the influence of unlabelled data on training after the model has basic discrimination ability, to ensure the stability of the training process.

[0032] (13) In the training process, the double model structure is adopted, which can continuously utilize unlabelled data in each training iteration, improve the utilization rate of training samples. At the same time, the stable target generated by the teacher model can suppress the performance fluctuation of the student model caused by error accumulation, thereby improving the robustness and generalization ability of the whole training process.

[0033] (14) The application provides a UI element intelligent classification method based on semi-supervised learning, which realizes stable and efficient model training by constructing a small-scale high-quality labeled data set, introducing a Mean Teacher framework and realizing consistency regularization constraint, thereby effectively improving the identification and grouping ability of fragment elements and overlapping elements in a UI design prototype and providing a solid foundation for automatic interface code generation. BRIEF DESCRIPTION OF DRAWINGS

[0034] The technical solutions of the application are further described below using examples and specific embodiments. In addition, some drawings are also used in the description of the technical solutions. For those skilled in the art, other drawings and the intention of the application can also be obtained from these drawings without creative labor.

[0035] Figure 1 The method flowchart of the embodiment of the application is shown in the figure. Figure 2 The training flowchart of the embodiment of the application based on the Mean Teacher semi-supervised learning framework is shown in the figure. Figure 3 The classification performance comparison experimental result diagram of the optimization model and the benchmark model proposed in the embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0036] In order to facilitate those skilled in the art to understand and implement the application, the application is further described in detail below in combination with the drawings and examples. It should be understood that the implementation examples described herein are only used to illustrate and explain the application, and are not used to limit the application.

[0037] In the process of automatically converting a UI design prototype into front-end code, the following technical problems exist: on the one hand, there are a large number of fragmented and overlapped UI elements in the prototype, which leads to a blurred logical structure between elements and seriously affects the accuracy of subsequent code generation; on the other hand, the existing element classification method mainly relies on a large amount of labeled data, and has high training cost, poor adaptability and a significant decrease in accuracy in a small sample scene, which limits its popularization in practical engineering applications.

[0038] Therefore, in view of the above problems, the embodiment of the application provides a user interface element intelligent classification method, system and product based on semi-supervised learning, which combines a small amount of high-quality artificial labeled data and a large amount of unlabeled prototype samples, constructs a deep neural network model with visual understanding ability and structure modeling ability, and realizes efficient identification and accurate grouping of UI elements.

[0039] The overall inventive concept of the embodiment is as follows: 1) Unified acquisition of image, text, color, position and category information of UI elements through multi-modal feature extraction mechanism, construction of input vector containing semantic and visual double characteristics; 2) Construction of deep learning model, introduction of Transformer architecture for context modeling of element sequence, capture of spatial and semantic dependency between elements in UI; training of deep learning model using Mean Teacher semi-supervised learning framework; 3) Output of classification label of each UI element, and automatic completion of element grouping process based on "start merge-merge-non-merge" label combination strategy; 4) Mapping of element grouping result to prototype, used for interface structure recovery or code generation.

[0040] Please refer to Figure 1 The embodiment provides a user interface element classification and grouping method based on semi-supervised learning, which comprises the following steps: Comprising the following steps: Step 1: Obtain UI elements, and convert multi-modal attributes of each UI element into embedded vectors in a unified format as input of a deep learning model; In an embodiment, in order to accurately classify user interface (UI) elements, the multi-modal attributes of each UI element are converted into embedded vectors in a unified format as input of a deep learning model. This stage includes the following sub-steps: Step B1, image feature embedding. This step extracts features from the image part of each UI element, and the extraction method uses a convolutional neural network model. Considering that the traditional image processing method often cuts image blocks with a bounding box, this method may introduce redundant background areas, causing visual redundancy between overlapping elements and affecting classification accuracy. To solve this problem, the invention uses an accurate alignment method, which encodes each element image after individual cutting to avoid interference information. In the model structure, a pre-trained ResNet-50 is preferably used as the image coding basic network. To adapt to the multi-class classification task of the invention, the output dimension of the last fully connected layer of the original model is modified to 256 to generate a fixed-dimensional visual embedding vector. All element images are uniformly scaled to a fixed size and processed by ResNet-50, outputting a two-dimensional tensor , each row corresponding to the image encoding of a UI element, where represents the number of UI elements, represents the embedding dimension of each element.

[0041] ​Step B2, text description embedding. The text description in UI elements often contains key function tips (such as button names, label texts, etc.), which are important for users to understand the interaction semantics. This step converts the text content into a dense vector suitable for neural network processing. First, the text is standardized, including converting all to lowercase, removing special characters, etc. In the selection of the tokenizer, considering that the RoBERTa model has stronger performance than the BERT in processing complex language structures, multi-language environments and professional vocabulary, the pre-trained RoBERTa tokenizer is used for tokenization, and it is mapped to a 32-dimensional index vector representation. To ensure the consistency of the input length, padding and truncation strategies are used to standardize all texts into fixed-length tensors. To align the dimensions of the text and image features, a compression strategy is used to sum each column of the tensor, compressing the first dimension of the tensor, and finally outputting a two-dimensional tensor .

[0042] Step B3, color and bounding box embedding. In interface design, color and spatial position are often closely related to element functions, such as button color, input box outline, etc. Therefore, the background color and bounding box coordinates of the element are modeled as independent features. Color information is represented in the channel, and each channel is normalized to the interval. If an element lacks color information, it is automatically filled with zeros as its default vector. The bounding box information contains the absolute position and size of the element in the canvas, and is represented in the format , and is scaled to the standard range by the min-max normalization method. Color embedding and frame embedding are processed through fully connected layers, so that their output shapes are .

[0043] Step B4, class attribute embedding. UI elements usually have pre-defined type identifiers, such as "button", "text box", "picture", "icon", etc. These types play an important guiding role in determining whether an element should be classified into a merged group. For example, text elements have semantic integrity and should not be split, so the model should learn to recognize the uniqueness of such elements. The element class label is input into the embedding layer to generate a dimensional class embedding vector , which enhances the model's understanding of structural semantics.

[0044] Step B5, multi-modal feature fusion output. This step embeds the five features of image, text, color, bounding box and class into equal-length dimensional vectors and performs a weighted element-by-element fusion on each vector to form a unified element-level embedding representation. This representation simultaneously encodes multiple features, including visual, semantic, structural, and layout, and serves as the input sequence for the subsequent Transformer encoder. Through this multimodal fusion mechanism, the model can achieve robust modeling of fragmented and overlapping UI elements while maintaining structural integrity, providing a guarantee for improved classification accuracy.

[0045] Step 2: Input the embedding vector in a unified format into the deep learning model and output the UI element classification results; The deep learning model consists of a position encoder, a multi-layer Transformer encoder and a SoftMax layer; the multi-layer Transformer encoder is a stacked model consisting of N Transformer encoder blocks of the same structure, each Transformer encoder block contains a multi-head self-attention mechanism layer MHA, a first normalization layer LN, a feedforward neural network FFN and a second normalization layer LN; wherein the output of the position encoder is connected to the output residual of the first normalization layer LN and then input into the feedforward neural network FFN; the input of the feedforward neural network FFN is connected to the output residual of the second normalization layer LN and then input into the SoftMax layer to output the classification result; N is a preset value; In one embodiment, after completing the multimodal feature extraction of UI elements, it is necessary to further perform structural modeling and classification on these elements to determine whether they belong to a merged group and their position within the group. This stage includes the following sub-steps: Step C1: Input sequence construction and position encoding. First, the multimodal features of each UI element (including image, text, bounding box, color and category embedding) are fused in a unified vector space to generate a dimension of The element embedding representation of . First, the image of each element in the sequence is converted into a dimensional visual embedding vector Then embed the various attributes of the element into In the dimensional space, the corresponding attribute embedding matrix (i.e. In order to enhance the model's ability to perceive the relative positions of elements in the interface, a position encoding strategy based on trigonometric functions of different frequencies is further adopted to convert the position encoding vector Added to the element features to generate the final embedding sequence containing position information: (1) Step C2: Input the embedded sequence into the deep learning model to generate the final classification representation ; For the first blocks, whose output is represented as ,and Represents the hidden state of this sublayer. In the initial stage, the element embedding is fed into the first Transformer encoder block. After N consecutive encoder blocks are processed, the final hidden state is obtained. Finally, the features are further mapped and transformed through a two-layer feed-forward network FFN (Feed-Forward Network, FFN) to generate the final classification representation : (2) (3) (4) Represents the hidden state of the kth position in the final Nth layer output Do feedforward mapping to generate representations for classification ; based on The function outputs the model Mapped into a probability distribution Each row of the probability distribution corresponds to the classification probability of an element, reflecting the probability that the element belongs to each of the three categories. Specifically, OK The output is represented as ,in Represents the classification label, which describes the The cross entropy loss function is used to evaluate the difference between the probability distribution predicted by the model and the actual label.

[0046] Step 3: Based on the classification results, group the output.

[0047] In an embodiment, after the classification is completed, each UI element will be assigned an explicit category label indicating its role in the potential combination relationship. To avoid the problem of poor adaptability of traditional rule-driven grouping methods in different design contexts, the present application proposes a label-driven grouping strategy that can automatically identify the structural unit to which the fragmented element belongs without relying on predefined heuristic rules. Specifically, the grouping process starts with an element labeled as "start-merge", and sequentially incorporates subsequent elements labeled as "merge" into the current grouping until the next "start-merge" label is encountered, at which point a new grouping unit is initiated. During this process, all elements labeled as "non-merge" are excluded from any grouping and are kept separately. This method ensures accurate aggregation of fragmented components while maintaining good generality and scalability.

[0048] In an embodiment, the deep learning model is a trained model. The training process includes the following steps: Step A: Constructing a UI prototype dataset Before training the user interface (UI) element classification model, the present application first constructs a high-quality UI prototype dataset and performs necessary data annotation processing to meet the subsequent requirements of multi-modal input and end-to-end classification. This stage includes the following sub-steps: Step A1, UI prototype data acquisition and format parsing. This step acquires Sketch format UI design prototype files in bulk from the OurSketch website, an open-source platform on the Internet. Sketch is a structured design tool, and its prototype files have a clear layer nesting structure, making them suitable for element-level data extraction. The prototype files are decompressed and parsed one by one to extract element information for each page, including position, size, color, content, category, and other attributes. This process is achieved through the SketchTool tool, which is a command-line interface tool provided by Sketch. It can export Sketch documents to ZIP format and automatically parse them into multiple JSON structure files, allowing quick access to layer structures, attribute data, and image resources. Through this tool, structured information can be extracted directly from Sketch files, bypassing the visual software interface, greatly improving batch processing efficiency.

[0049] Step A2, element image and attribute information extraction. To construct a complete UI element dataset, image and attribute extraction is needed for each visual element in the prototype file. This step uses a depth-first traversal algorithm of tree structure to expand the layer hierarchy level by level, from the top page to each nested layer, ensuring that no interface elements are missed. Each traversed element exports its corresponding image snapshot, stacking structure, and JSON data containing detailed attributes. These attributes include but are not limited to: element coordinates, size, layer order, background color, text content, font style, etc. Each element data after extraction consists of three files: one is the prototype file, which shows the overall interface; the second is the element stacking diagram file, which shows the layer snapshot of a single element; the third is the element attribute file, which stores structured description information. This three-in-one data format ensures comprehensive modeling of visual and semantic information in the classification task.

[0050] Step A3, effective sample screening and data cleaning. Due to problems such as abnormal layer structure, missing image files, and non-standard JSON fields in some Sketch prototypes, the invention sets a series of screening rules for effective sample screening after preliminary parsing. For example, integrity checking, outlier filtering, logical conflict processing, and invalid element removal. After this cleaning process, 1230 UI prototype data that meet the format standards and content integrity are successfully screened from the initial 1000+ prototypes, forming a high-quality raw dataset for subsequent training.

[0051] Step A4, professional annotation and label strategy design. Considering the high sensitivity of deep learning models to label accuracy, especially when dealing with fragmented and overlapping elements, the requirement for label granularity and semantic consistency is higher, therefore the invention uses artificial fine labeling to construct the label set. This step selects 100 prototype pages with clear structure and diverse styles, and invites professionals with front-end development background to perform element-by-element manual annotation. To implement subsequent grouping operations, each element is assigned one of the three labels: "start merge", "merge", and "non-merge". The annotation strategy is as follows: for each group of elements that should be merged, the first appearing element is marked as "start merge" and the rest are marked as "merge"; if the element does not belong to any merged group, it is marked as "non-merge". All elements are initially in the "non-merge" state and are modified by the annotator according to semantic and visual logic as needed. This method not only guarantees the consistency and reproducibility of the annotation, but also provides clear boundaries for grouping and classification.

[0052] Step A5, label quality review and data quality control. To further ensure the reliability of the labeled data, the present application sets up a special manual review link after completing the preliminary labeling. The reviewer reviews and evaluates the labels of the prototype according to the preset rules. If conflicts, inconsistencies or omissions are found in the review process, they are fed back to the labeler for revision. Through double manual processing, it ensures that the label data has high accuracy, high consistency and good generalization, providing a solid data foundation for subsequent semi-supervised training and model evaluation.

[0053] In a specific implementation process, the data set used contains the following three data sets: Data set A (strong supervision): 100 labeled data are divided into 80:20 training set and test set, which are used to compare the baseline method; Data set B (small semi-supervised): only 1 / 8 of the 80 training samples is used as labeled data, and the rest is considered as unlabeled; Data set C (large-scale semi-supervised): 80 of the 100 prototypes are used as labeled training set, and 900 unlabeled data are used as auxiliary training.

[0054] Table 1 strong supervision learning data set A used in the embodiment

[0055] Table 2 small semi-supervised learning data set B used in the embodiment

[0056] Table 3 semi-supervised learning data set C used in the embodiment

[0057] Step B: convert the multi-modal attributes of each UI element into embedded vectors in a unified format, as the input of the deep learning model, and use the semi-supervised training method of the Mean Teacher framework to train the deep learning model; In an embodiment, the deep learning model backbone is a 6-layer Transformer encoder with 8 attention heads and an MLP hidden dimension of 2048. The multi-modal embedded vectors are constructed into an element sequence in page order and input into the multi-layer Transformer encoder after adding position encoding to learn the context structure between elements.

[0058] In an embodiment, please see Figure 2 , the semi-supervised training method of the Mean Teacher framework, specifically including the following sub-steps: Step B1, double model structure initialization. The semi-supervised learning system in this embodiment consists of two models with shared parameters but different updating strategies: the student model (Student) and the teacher model (Teacher). Both models use the same UI element classification structure in the initial stage and achieve initial parameters through parameter copying.

[0059] The parameters of the teacher model are not directly trained through backpropagation, but are updated through the Exponential Moving Average (EMA) of the student model parameters, keeping the teacher model as a stable target for the student model prediction.

[0060] Step B2, teacher model parameter updating mechanism. During the training process, the student model updates its parameters ( representing the trainable parameters in the student model) after each iteration; at the same time, the parameters of the teacher model are updated smoothly according to the following formula: (5) where represents the parameters of the model in the th iteration, represents the parameters of the model in the th iteration, and represents the EMA decay coefficient, which determines the "inertia" of the model parameters and controls the weighted average of new and old parameters; the value of a gradually increases with the training round, and the specific update is: (6) This dynamic adjustment mechanism ensures that the teacher model changes rapidly in the early training to quickly respond to the progress of the student model, and remains stable in the later training to provide a reliable learning target for the student model.

[0061] Step B3, training with labeled data (supervised learning). For labeled samples , this embodiment introduces data augmentation noise to obtain perturbed samples , and inputs them into the student model for prediction to generate output . Then the predicted results are compared with the true labels to calculate the cross-entropy loss : (7) where, ​​It is a classification loss function used to measure the difference between the model's predicted probability distribution and the true label, and optimizes the model parameters by minimizing the negative log-likelihood. This supervised loss is used to guide the student model to learn the correct element classification boundaries.

[0062] Step B4: Unlabeled data training (consistency regularization). To utilize unlabeled data , this embodiment injects different noises into it ( ), generate perturbation samples The sample is input into the student model and the teacher model respectively to generate the prediction results and The consistency loss is obtained by calculating the mean square error (MSE) of the two output results. : (8) in, Represents the number of unlabeled data samples in the current batch. This loss encourages the student model to maintain consistent predictions with the teacher model in the face of input perturbations, thereby enhancing the robustness of the model.

[0063] Step B5: Construction of total loss function and weight adjustment. and consistency loss Weighted fusion to construct the total training loss function Loss: (9) in, For the weight of the consistency loss, the following formula is used to implement a dynamic adjustment mechanism that grows with the training progress: (10) (11) This weight adjustment strategy can control the consistency signal to slowly increase in the early stage of training, and gradually increase the influence of unlabeled data on training after the model has basic discrimination ability, ensuring the stability of the training process.

[0064] In one embodiment, the Adam optimizer is used for training, with an initial learning rate of 1e-4, which is decayed to 1 / 10 of its original value after every 200 training epochs. Dropout of 0.2 and L2 regularization are used to reduce the risk of overfitting. The batch size is 10, including 4 labeled samples and the remaining unlabeled samples. All embedding dimensions are uniformly set to 256, and the Transformer encoder contains 6 layers and 8 attention heads. The entire training is performed using the PyTorch framework on a device configured with an RTX3090 GPU, an Intel i7 CPU, and 32GB of memory, for a total of 300 epochs.

[0065] Step B6, parameter updating and optimization strategy. During training, the student model parameters are updated through standard backpropagation and gradient descent, while the teacher model parameters are updated through EMA without backpropagation. This dual model structure can continuously utilize unlabeled data in each training iteration, improving the utilization of training samples. At the same time, the stable target generated by the teacher model can suppress the performance fluctuations caused by error accumulation in the student model, thereby improving the robustness and generalization ability of the overall training process.

[0066] The present embodiment shows superior performance in three data set settings (strong supervision, small sample semi-supervision, and large-scale semi-supervision), as shown in Figure 3 Compared with the EfficientNet, Swin Transformer, and EGFE baseline methods on dataset A, the results show that the present method is superior to the above models in precision, recall, and F1 score, verifying its performance advantage in strong supervision scenarios. In semi-supervised experiments on dataset B and dataset C, compared with the baseline model trained entirely based on strong supervision, the present method improves the precision, F1, and recall by 17.2%, 16.4%, 15.3% (dataset B) and 3.3%, 2.8%, 2.2% (dataset C), respectively, indicating that the semi-supervised strategy significantly improves the classification performance under the condition of small samples.

[0067] Under the condition of a small amount of labeled data (such as using only 100 labeled prototype graphs), the present method significantly improves the precision, recall, and F1 score compared with the baseline method, with a maximum improvement of more than 15%. After further introducing a large amount of unlabeled data, the performance of the present model continues to improve, proving the effectiveness of the semi-supervised strategy in complex UI element recognition scenarios.

[0068] The present method not only improves the classification accuracy but also significantly reduces the dependence on labeled data, which is helpful for the stable execution of the automatic interface restoration and code generation process.

[0069] The present embodiment also provides a UI element intelligent classification system based on semi-supervised learning, which includes: one or more processors; a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the UI element intelligent classification method based on semi-supervised learning.

[0070] The present embodiment also provides a UI element intelligent classification product based on semi-supervised learning, which includes computer program instructions that, when executed on a computer, cause the computer to perform the UI element intelligent classification method based on semi-supervised learning.

[0071] It should be understood that the above-described embodiments are part of the embodiments of the present application, but not all the embodiments. In addition, the technical features of each embodiment or individual embodiments provided by the present application can be combined with each other to form a feasible technical solution, and such combination is not restricted by the order of steps and / or structure mode, but must be based on the realization of ordinary skilled in the art, when the combination of technical solutions appears contradictory or unfeasible, it should be considered that such combination of technical solutions does not exist, nor within the scope of protection required by the present application.

[0072] It should be understood that the above description of the preferred embodiments is more detailed, and therefore should not be considered as a limitation on the scope of patent protection of the present application. Those skilled in the art can make substitutions or modifications without departing from the scope of protection claimed by the present application, which falls within the scope of protection of the present application. The scope of protection of the present application shall be subject to the appended claims.

Claims

1. A UI element intelligent classification method based on semi-supervised learning, characterized in that: The following steps are involved: Step 1: Obtain UI elements and convert the multimodal attributes of each UI element into an embedded vector in a unified format as input to the deep learning model; Step 2: Input the embedding vector in a unified format into the deep learning model and output the UI element classification results; The deep learning model consists of a position encoder, a multi-layer Transformer encoder and a SoftMax layer; the multi-layer Transformer encoder is a stacked model consisting of N Transformer encoder blocks of the same structure, each Transformer encoder block contains a multi-head self-attention mechanism layer MHA, a first normalization layer LN, a feedforward neural network FFN and a second normalization layer LN; wherein the output of the position encoder is connected to the output residual of the first normalization layer LN and then input into the feedforward neural network FFN; the input of the feedforward neural network FFN is connected to the output residual of the second normalization layer LN and then input into the SoftMax layer to output the classification result; N is a preset value; Step 3: Based on the classification results, group the output.

2. The method for intelligent classification of UI elements based on semi-supervised learning according to claim 1 is characterized in that: The specific implementation of step 1 includes the following sub-steps: Step 1.1: Use a convolutional neural network model to extract the features of the image part of each UI element and obtain the image embedding vector ; Step 1.2: Convert the text content of each UI element into a dense vector suitable for convolutional neural network processing to obtain the text embedding vector ; Step 1.3: Extract the background color and bounding box coordinates of the element to obtain the color embedding vector and the bounding frame embedding vector ; Step 1.4: Input the element category label into the embedding layer to generate dimensional category embedding vector ; Step 1.5: Embed the five features of image, text, color, bounding box and category into equal length dimensional vectors, and performs element-wise weighted fusion on each vector to form a unified element-level embedding representation.

3. The method for intelligent classification of UI elements based on semi-supervised learning according to claim 2, characterized in that: In step 1.1, the image of each UI element is cropped and encoded separately using precise alignment. Then, the pre-trained ResNet-50 is used as the image encoding basic network. The output dimension of the last fully connected layer of the ResNet-50 is Set to 256 to generate a fixed-dimensional visual embedding vector; all UI element images are uniformly scaled to a fixed size and processed by ResNet-50 to output a two-dimensional tensor , each row corresponds to the image code of a UI element; Indicates the number of UI elements, Represents the embedding dimension of each element.

4. The method for intelligent classification of UI elements based on semi-supervised learning according to claim 2, characterized in that: In step 1.2, the text is first normalized, including converting all to lowercase and removing special characters; then the pre-trained RoBERTa tokenizer is used for tokenization and mapped into a 32-dimensional index vector representation; further padding and truncation strategies are used to normalize all text into a fixed-length tensor; in order to align the text features with the image feature dimensions, a compression strategy is finally used to sum each column of the tensor, compress the first dimension of the tensor, and finally output a two-dimensional tensor. .

5. The method for intelligent classification of UI elements based on semi-supervised learning according to claim 2, characterized in that: In step 1.3, the color information is Channel representation, and normalize each channel to Interval; if an element lacks color information, it will be automatically padded with zero as its default vector; The bounding box information contains the element's absolute position within the canvas ( ) and size , expressed in the format , and scaled to the standard range using the minimum-maximum normalization method, where Represents length and width respectively; color embedding vector and the bounding frame embedding vector After processing through the fully connected layer, their output shapes are ;in, Indicates the number of UI elements, Represents the embedding dimension of each element.

6. The method for intelligent classification of UI elements based on semi-supervised learning according to claim 1, characterized in that: The specific implementation of step 2 includes the following sub-steps: Step 2.1: Use a position encoding strategy based on trigonometric functions of different frequencies to encode the position encoding vector Added to the element features to generate the final embedding sequence containing position information; ； in, Represents image embedding vector, color embedding vector, category embedding vector, bounding frame embedding vector, and text embedding vector respectively; Step 2.2: Input the embedded sequence into the deep learning model to generate the final classification representation ; For the first blocks, whose output is represented as ,and Represents the hidden state of the sub-layer; in the initial stage, the element embedding is fed into the first Transformer encoder block; after N consecutive encoder blocks, the final hidden state is obtained Finally, the features are further mapped and transformed through a two-layer feedforward network FFN to generate the final classification representation ; ； ； ； Represents the hidden state of the kth position in the final Nth layer output Do feedforward mapping to generate representations for classification ; Step 2.3: Based on The function outputs the model Mapped into a probability distribution ; Each row of the probability distribution corresponds to the classification probability of an element, reflecting the probability that the element belongs to each of the three categories; OK The output is represented as ,in Represents the classification label, which describes the The predicted probability distribution of elements.

7. The method for intelligent classification of UI elements based on semi-supervised learning according to claim 1, characterized in that: In step 3, grouping is performed based on the label-driven grouping strategy and then output; The grouping process starts with the element marked as "Start Merge" and sequentially assigns the subsequent elements marked as "Merge" to the current group until the next "Start Merge" tag is encountered, at which time a new grouping unit is started; during this process, all elements marked as "Non-Merge" will be excluded from any grouping and retained separately.

8. The method for intelligent classification of UI elements based on semi-supervised learning according to any one of claims 1 to 7, characterized in that: The deep learning model is a trained model; The training process consists of the following steps: Step A: Construct UI prototype image dataset; Step A1: UI prototype data collection and format analysis; Obtain UI design prototype files in Sketch format in batches from the open source Internet platform OurSketch. Use SketchTool to decompress and parse the prototype files one by one, extracting element information for each page, including position, size, color, content, and category attributes. Step A2: element image and attribute information extraction; A tree-structured depth-first traversal algorithm is used to gradually expand the layer hierarchy, starting from the top-level page and drilling down to each nested layer. For each traversed element, its corresponding image snapshot, stacking structure, and JSON data containing detailed attributes are exported. These detailed attributes include, but are not limited to, element coordinates, dimensions, layer order, background color, text content, and font style. Each extracted element data consists of three files: a prototype file that displays the overall interface; an element stacking file that displays a layer snapshot of a single element; and an element attribute file that stores structured description information. Step A3: After completing the initial analysis, filtering rules are applied to ensure data quality, including integrity verification, outlier filtering, logical conflict handling, and invalid element removal. Integrity verification requires removing samples that are missing key files or attributes. Outlier filtering ensures that bounding box coordinates are normalized to the [0,1] interval and color channel values ​​are valid. Logical conflict handling involves manual review of samples with visual overlap exceeding 50% or inconsistent layer labels. Invalid element removal filters out decorative layers that are too small or too transparent. Step A4: A label set is constructed using manual precision labeling. Each element is assigned one of three labels: "starting merge," "merged," or "not merged." For each group of elements to be merged, the first element that appears is labeled "starting merge," and the remaining elements are labeled "merged." Elements that do not belong to any merge group are labeled "not merged." All elements are initially labeled "not merged," and are modified by human annotators based on semantics and visual logic. Step A5: Label quality review and data quality control; After the initial annotation is completed, manual review is conducted. The reviewer reviews and evaluates the quality of the prototype labels according to preset rules. If any label conflicts, inconsistencies, or omissions are found during the review process, they will be promptly fed back to the annotator for revision. Step B: Convert the multimodal attributes of each UI element into a unified embedding vector as input to the deep learning model. The deep learning model is trained using the semi-supervised training method of the Mean Teacher framework. The semi-supervised training method of the Mean Teacher framework includes two models with shared parameters but different update strategies: a student model and a teacher model. Both models initially use the same UI element classification structure and are initialized through parameter replication. During training, the student model parameters are updated using standard backpropagation and gradient descent. The teacher model parameters are not directly trained via backpropagation, but are updated using an exponential moving average (EMA) of the student model parameters, maintaining the teacher model as a stable target for student model predictions. During training, the supervision loss and consistency loss Weighted fusion to construct the total training loss function Loss: ； in, For the weight of the consistency loss, the following formula is used to implement a dynamic adjustment mechanism that grows with the training progress: ； ； in, is the maximum value of the consistency weight, It is the number of cycles that the consistency weight gradually increases. It is used to control the growth rate of the consistency weight in the early stage of training to ensure that the consistency weight grows slowly in the early stage of training and gradually accelerates as the training progresses until it reaches the maximum value. Indicates the number of current training iterations, Function is used as activation function to achieve smooth increase; For labeled samples , introducing data augmentation noise , get the perturbation sample , and input it into the student model for prediction, generating output ; Then compare the predicted results with the true labels Calculating cross entropy loss : ； in, It is a classification loss function used to measure the difference between the model's predicted probability distribution and the true label, and optimizes the model parameters by minimizing the negative log-likelihood; For unlabeled data , inject different noises into it , generate perturbation samples ; The sample is input into the student model and the teacher model respectively to generate the prediction results and ; Obtain consistency loss by calculating the mean square error MSE of the two output results : ； in, Indicates the number of samples of unlabeled data in the current batch; during the training process, the student model will update its parameters after each iteration , Represents the trainable parameters in the student model; at the same time, the parameters of the teacher model Smooth update is performed according to the following formula: ； Among them, express The model is in Parameters in round iterations, express The model is in Parameters in round iterations; Represents the EMA decay coefficient, which determines The "inertia" of the model parameters is used to control the weighted average of the new parameters and the old parameters; the value of α varies with the number of training rounds. Gradually increase, the specific updates are: 。 9. A UI element intelligent classification system based on semi-supervised learning, characterized by: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the UI element intelligent classification method based on semi-supervised learning as described in any one of claims 1 to 8.

10. A UI element intelligent classification product based on semi-supervised learning, comprising computer program instructions, characterized in that: When the computer program instructions are executed on a computer, the computer is enabled to execute the UI element intelligent classification method based on semi-supervised learning according to any one of claims 1 to 8.

Images