A UI design diagram segmentation method based on front-end semantic understanding and related equipment

Abstract

The application discloses a UI design graph segmentation method based on front-end semantic understanding and related equipment, and the method comprises the following steps: obtaining a UI design graph to be segmented, pre-processing the UI design graph to obtain a processed design graph; based on the processed design graph, performing feature extraction by using a pre-trained multi-modal deep neural network to obtain multi-modal features; based on the multi-modal features, performing component semantic segmentation on the processed design graph to obtain component data; based on original boundaries and component categories, performing rule clustering on atomic-level components to construct a multi-level semantic hierarchy system; and outputting a UI design graph segmentation result according to the multi-level semantic hierarchy system. The application can effectively realize semantic organization from atomic components to composite components, overcomes the problems that the prior art has poor adaptability to complex UIs and is difficult to capture implicit semantics, and can be widely applied to the technical field of image processing.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a UI design drawing segmentation method and related equipment based on front-end semantic understanding. Background Technology

[0002] Although existing UI design drawing segmentation methods have made some progress in visual recognition and segmentation, they generally suffer from the following technical problems, which limit their effectiveness and intelligence level in practical applications:

[0003] (1) Lack of deep semantic understanding: Most existing methods are limited to visual feature recognition and physical area division, making it difficult to truly understand the "meaning" and "function" of UI elements. For example, they can identify a rectangular area as a "button", but cannot understand whether the button is used to "submit a form" or "jump to a page", nor can they distinguish the semantic differences between two visually similar buttons (such as the "login" button and the "register" button). This lack of semantic understanding results in a low level of intelligence in the segmentation results, making it difficult to meet higher-level automation needs, such as intelligent code generation and intelligent UI test case generation.

[0004] (2) Poor adaptability to complex and diverse UIs: Existing methods often experience a significant decrease in segmentation accuracy and robustness when faced with UI designs that are diverse in style, complex in layout, and contain a large number of custom components. This is because purely visual feature-based methods struggle to capture the implicit logical and semantic information in UI designs, which is crucial for correctly segmenting complex UIs. For example, a navigation bar may visually appear in multiple forms (horizontal, vertical, folded, etc.), and existing methods may require separate training or rule definition for each form, making it difficult to achieve universality.

[0005] (3) Difficulty in handling non-explicit semantic information: The semantics of many UI elements are not entirely determined by their visual appearance, but may also depend on context, text content, and even potential interactive behaviors. Existing methods often struggle to effectively utilize this non-explicit semantic information for segmentation. For example, the semantics of a text input box (search box, username input box, comment box) often depends on the label or placeholder text next to it, rather than just its visual shape. Existing methods often require additional manual annotation or complex post-processing when handling such cases, increasing workload and error rate.

[0006] (4) Mismatch between the granularity and semantics of the segmentation results: Existing methods may only be able to perform coarse-grained region segmentation (such as dividing the page into header, body, and footer) or fine-grained element recognition (such as recognizing a single button), but it is difficult to flexibly switch between different granularities and ensure that the segmentation results are highly matched with the semantic structure of the UI. For example, a "product list" area may contain multiple "product cards", and each card contains elements such as "product image", "product name", and "price". Existing methods may find it difficult to identify the semantic region of "product list" at once, and at the same time identify its internal semantic sub-regions and elements. Summary of the Invention

[0007] The main objective of this invention is to propose a UI design diagram segmentation method, device, electronic device, storage medium, and program product based on front-end semantic understanding, aiming to solve at least one problem of the prior art.

[0008] To achieve the above objectives, one aspect of this invention proposes a UI design diagram segmentation method based on front-end semantic understanding, the method comprising:

[0009] Obtain the UI design draft to be segmented, preprocess the UI design draft, and obtain the processed design draft;

[0010] Based on the processing design graph, feature extraction is performed using a pre-trained multimodal deep neural network to obtain multimodal features; among which, multimodal features include visual features, text features, and layout features;

[0011] Based on multimodal features, component semantic segmentation is performed on the processing design graph to obtain component data; the component data includes the original boundary and component category of each atomic-level component in the processing design graph;

[0012] Based on the original boundaries and component categories, atomic-level components are clustered according to rules to construct a multi-level semantic hierarchy system.

[0013] The UI design diagram segmentation results are output based on a multi-level semantic hierarchy system.

[0014] In some embodiments, preprocessing of the UI design draft includes at least one of the following steps:

[0015] Based on the target size, the UI design draft is scaled up, and the mapping relationship between the original size and the target size of the UI design draft is recorded;

[0016] Convert UI design drafts to a preset color space;

[0017] Image noise removal is performed on UI design drafts using Gaussian filtering;

[0018] Sharpen the blurred areas in the UI design draft;

[0019] Perform grayscale normalization on the UI design drafts.

[0020] In some embodiments, when the result of feature extraction is visual features, feature extraction is performed using a pre-trained multimodal deep neural network, including the following steps:

[0021] The convolutional layers of the pre-trained visual encoder are loaded as the original visual model. The convolutional layers in the original visual model are frozen to initialize the initial visual model. The freezing operation includes the first number of convolutional layers before freezing and the last second number of convolutional layers. The sum of the first number and the second number is the total number of convolutional layers in the initial visual model.

[0022] The initial visual model is trained using a design drawing labeled with visual features. The average classification accuracy is constructed based on the training feature map output by the penultimate convolutional layer in the initial visual model and the visual feature labels. The parameters of the initial visual model are adjusted based on the average classification accuracy to obtain the target visual model.

[0023] The processing design diagram is input into the target visual model. The target feature map output by the penultimate convolutional layer in the target visual model is reduced in dimensionality and encoded by global average pooling and L2 normalization to obtain visual features.

[0024] In some embodiments, when the result of feature extraction is text features, feature extraction is performed using a pre-trained multimodal deep neural network, including the following steps:

[0025] The design drawing is processed using a pre-set optical character recognition engine to perform text recognition, resulting in the recognized text.

[0026] The bounding box of the text region corresponding to each recognized text is located using a multi-scale text detection algorithm.

[0027] Recurrent neural networks or transformer models are used to identify the text content within the bounding boxes of text regions to obtain structured text;

[0028] Structured text is input into a pre-trained large language model, and text semantic embeddings are obtained as text features through forward propagation of the model.

[0029] In some embodiments, when the result of feature extraction is layout features, feature extraction is performed using a pre-trained multimodal deep neural network, including the following steps:

[0030] Once the HTML source code of the UI design draft is obtained, layout features based on the DOM structure are extracted from the design draft.

[0031] The layout feature extraction based on the DOM structure includes the following operations:

[0032] The DOM tree structure is built based on the HTML source code; the DOM tree structure includes the element hierarchy of all DOM elements.

[0033] The bounding box coordinates of each DOM element are obtained by converting CSS style information from the HTML source code.

[0034] Key elements are matched and identified in the DOM by tag name and / or class name, and layout features are generated based on the key elements. The layout features include the bounding box coordinates of the key elements, as well as the hierarchical relationship and the number of sibling elements determined by the element hierarchy.

[0035] When the HTML source code of the UI design is not obtained, the layout features of the design are inferred based on visual analysis.

[0036] The layout feature inference based on visual analysis includes the following operations:

[0037] The design drawing is divided into text and non-text regions by combining optical character recognition with image segmentation algorithms.

[0038] Each text region is treated as a text element, and the bounding box coordinates of the text region are output based on the results of optical character recognition.

[0039] Contour detection is performed on non-text regions, and weak edges are strengthened through adaptive thresholding and edge enhancement to obtain non-text elements and their corresponding bounding box coordinates.

[0040] Based on the vertical and overlapping relationships between text and non-text elements in the design drawing, the element hierarchy is obtained.

[0041] Layout features are generated based on bounding box coordinates and element hierarchy.

[0042] In some embodiments, component semantic segmentation is performed on the processing design graph based on multimodal features to obtain component data, including the following steps:

[0043] A semantic segmentation network is used to process and obtain the atomic-level components and their corresponding original boundaries in the processing design diagram.

[0044] A lightweight classifier is used to determine the semantic label of the detection region corresponding to each original boundary, thereby obtaining the component category of each atomic component.

[0045] In some embodiments, rule-based clustering of atomic-level components is performed to construct a multi-level semantic hierarchy, including the following steps:

[0046] Atomic-level components are treated as fine-grained regions at the component level, the original boundaries are treated as component-level bounding boxes, and component categories are treated as component-level semantic labels.

[0047] Based on the original boundaries, the spatial relationships between each atomic-level component are determined, and then component clustering based on component categories is performed to obtain module-level medium-granularity regions.

[0048] Based on the region range labeling module-level bounding boxes corresponding to the medium-granularity regions, semantic label determination is performed on the detection regions corresponding to the module-level bounding boxes to obtain the module-level semantic label for each medium-granularity region.

[0049] By using rule templates based on page layout logic, the design draft is divided into multiple large areas, and each medium-granular area is classified into the corresponding large area to obtain page-level coarse-granular areas.

[0050] Page-level bounding boxes are marked based on the region range corresponding to the coarse-grained regions. Semantic labels are determined for the detection regions corresponding to the page-level bounding boxes to obtain page-level semantic labels for each coarse-grained region.

[0051] To achieve the above objectives, another aspect of the present invention proposes a UI design diagram segmentation device based on front-end semantic understanding, the device comprising:

[0052] The first module is used to obtain the UI design drawing to be segmented, preprocess the UI design drawing, and obtain the processed design drawing;

[0053] The second module is used to extract features based on the processing design graph using a pre-trained multimodal deep neural network to obtain multimodal features; among which, multimodal features include visual features, text features, and layout features;

[0054] The third module is used to perform component semantic segmentation on the processing design graph based on multimodal features to obtain component data; the component data includes the original boundary and component category of each atomic component in the processing design graph;

[0055] The fourth module is used to perform rule-based clustering of atomic-level components based on the original boundaries and component categories, thereby constructing a multi-level semantic hierarchy system.

[0056] The fifth module is used to output the UI design diagram segmentation results based on the multi-level semantic hierarchy.

[0057] To achieve the above objectives, another aspect of the present invention provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned method.

[0058] To achieve the above objectives, another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0059] To achieve the above objectives, another aspect of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method.

[0060] The embodiments of the present invention include at least the following beneficial effects: The present invention provides a UI design drawing segmentation method, device, electronic device, storage medium, and program product based on front-end semantic understanding. This solution obtains a UI design drawing to be segmented, preprocesses the UI design drawing to obtain a processed design drawing; based on the processed design drawing, it extracts features using a pre-trained multimodal deep neural network to obtain multimodal features; wherein, the multimodal features include visual features, text features, and layout features; based on the multimodal features, it performs component semantic segmentation on the processed design drawing to obtain component data; wherein, the component data includes the original boundary and component category of each atomic-level component in the processed design drawing; based on the original boundary and component category, it performs rule-based clustering of the atomic-level components to construct a multi-level semantic hierarchy system; and outputs the UI design drawing segmentation result according to the multi-level semantic hierarchy system. This invention significantly improves the semantic understanding of UI elements by introducing a multimodal deep neural network and integrating visual, textual, and layout features for joint analysis. It can accurately identify the functional meaning of atomic-level components. Furthermore, by constructing a multi-level semantic hierarchy through rule-based clustering, it effectively achieves semantic organization from atomic components to composite components, overcoming the problems of poor adaptability to complex UIs and difficulty in capturing implicit semantics in existing methods. Specifically, this invention can parse non-explicit semantic information based on context and text content, and achieve structural and semantic matching segmentation at different granularities, significantly improving the accuracy and usability of the segmentation results and providing a reliable foundation for advanced UI applications. Attached Figure Description

[0061] Figure 1 This is a schematic diagram of an implementation environment for the UI design diagram segmentation method based on front-end semantic understanding provided in this embodiment of the invention;

[0062] Figure 2 This is a flowchart illustrating a UI design diagram segmentation method based on front-end semantic understanding provided in an embodiment of the present invention;

[0063] Figure 3 This is a schematic diagram illustrating an example of academic search webpage input provided in an embodiment of the present invention;

[0064] Figure 4 This is a schematic diagram illustrating an example of news search webpage input provided in an embodiment of the present invention;

[0065] Figure 5 This is a schematic diagram of the unfolding process of step S300 provided in an embodiment of the present invention;

[0066] Figure 6 This is a schematic diagram of the overall process of the UI design diagram segmentation method based on front-end semantic understanding provided in the embodiments of the present invention;

[0067] Figure 7 This is a schematic diagram illustrating an example of news search webpage output results provided in an embodiment of the present invention;

[0068] Figure 8 This is a schematic diagram illustrating an example of academic search webpage output results provided in an embodiment of the present invention;

[0069] Figure 9 This is a schematic diagram of a UI design drawing segmentation device based on front-end semantic understanding provided in an embodiment of the present invention;

[0070] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0072] It is understood that the terms “first,” “second,” etc., used in this invention may be used herein to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. For example, first information may also be referred to as second information without departing from the scope of embodiments of the invention, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to determination” as used herein may be interpreted as “when…” or “when…” or “in response to determination.”

[0073] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0074] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0075] To facilitate understanding of the technical solution of this invention, the technical terms that may be applied in the technical solution of this invention will first be explained:

[0076] Front-end semantic understanding refers to the process of analyzing the visual elements, layout, interaction logic, and text content of a user interface (UI) to infer and identify the actual meaning and function of these elements. For example, identifying a button as a "submit" button, a text box as a "search" input box, or an area as a "navigation bar".

[0077] UI Design Image Segmentation refers to the process of automatically dividing a complete UI design image (such as a webpage screenshot or app interface diagram) into multiple areas with independent semantics or functions. These areas can be navigation bars, content areas, sidebars, footers, etc., or more granular components such as buttons, input boxes, and images. The purpose of segmentation is to better understand the structure and content of the UI, providing a foundation for subsequent code generation, automated testing, or design assistance.

[0078] Visual elements: The basic graphic units that make up a user interface, including but not limited to text, images, icons, buttons, input boxes, drop-down menus, check boxes, radio buttons, progress bars, sliders, etc.

[0079] Layout refers to the spatial arrangement and organization of visual elements in a UI design, including the position, size, spacing, and alignment of elements. Layout is an important clue for understanding the structure and semantics of a UI.

[0080] Interaction logic refers to the behaviors and responses triggered when a user interacts with UI elements. For example, clicking a button might redirect to a page or submit data, while entering text might trigger search suggestions. Understanding interaction logic helps in more accurately identifying the semantics of UI elements.

[0081] Semantic Region: A semantic region refers to an independent area in a UI design that has a specific function or meaning. For example, the header area of ​​a webpage typically contains a logo, navigation menu, and search box, and can be identified as a "header" semantic region; an area containing an article title, body text, and images can be identified as an "article content" semantic region.

[0082] Deep learning is a branch of machine learning that learns complex patterns in data by building and training deep neural networks. In UI design graph segmentation, deep learning models can be used to identify and classify visual elements, understand layout structures, and divide semantic regions.

[0083] Multimodal Large Model: A multimodal large model is a deep learning model capable of processing and generating multiple types of data (such as text, images, audio, etc.). By combining data from different modalities, these models can understand and generate richer and more complex information. For example, they can generate images from text descriptions or generate corresponding text descriptions from images.

[0084] Computer vision is a branch of artificial intelligence that aims to enable computers to "see" and understand images or videos. In UI design graph segmentation, computer vision techniques are used for image processing, feature extraction, and pattern recognition.

[0085] Image segmentation: A task in computer vision that aims to divide a digital image into multiple image segments (pixel groups). In UI design image segmentation, it specifically refers to dividing pixels in a UI design image into different semantic regions.

[0086] Object detection: A task in computer vision that aims to identify instances of specific objects in an image and label them with bounding boxes. In UI design graph segmentation, it can be used to identify various components in the UI.

[0087] UI automation refers to the technology of automatically simulating user actions to interact with the user interface through programs. UI design sketch segmentation can provide UI automation with more intelligent element recognition and location capabilities.

[0088] Among related technologies, although existing UI design drawing segmentation methods have made some progress in visual recognition and segmentation, they generally have technical limitations that restrict their effectiveness and intelligence level in practical applications.

[0089] In view of this, this invention provides a UI design diagram segmentation method and related device based on front-end semantic understanding. This method involves acquiring a UI design diagram to be segmented, preprocessing the UI design diagram to obtain a processed design diagram, extracting features from the processed design diagram using a pre-trained multimodal deep neural network to obtain multimodal features, including visual features, text features, and layout features, performing component semantic segmentation on the processed design diagram to obtain component data, including the original boundaries and component categories of each atomic-level component in the processed design diagram, performing rule-based clustering on the atomic-level components to construct a multi-level semantic hierarchy, and outputting the UI design diagram segmentation result based on the multi-level semantic hierarchy. This invention significantly improves the semantic understanding of UI elements by introducing a multimodal deep neural network and integrating visual, textual, and layout features for joint analysis. It can accurately identify the functional meaning of atomic-level components. Furthermore, by constructing a multi-level semantic hierarchy through rule-based clustering, it effectively achieves semantic organization from atomic components to composite components, overcoming the problems of poor adaptability to complex UIs and difficulty in capturing implicit semantics in existing methods. Specifically, this invention can parse non-explicit semantic information based on context and text content, and achieve structural and semantic matching segmentation at different granularities, significantly improving the accuracy and usability of the segmentation results and providing a reliable foundation for advanced UI applications.

[0090] It is understood that the UI design graph segmentation method based on front-end semantic understanding provided by this invention can be applied to any computer device with data processing and computing capabilities, and this computer device can be various terminals or servers. When the computer device in the embodiment is a server, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal can be a smartphone, tablet, laptop, or desktop computer, but it is not limited to these.

[0091] like Figure 1 The diagram shown is a schematic representation of an implementation environment provided by an embodiment of the present invention. (Refer to...) Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected via a network, either wirelessly or via a wired connection, to complete data transmission and exchange.

[0092] Server 101 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0093] Additionally, server 101 can also be a node server in a blockchain network. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0094] Terminal 102 can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. Terminal 102 and server 101 can be directly or indirectly connected via wired or wireless communication, and this embodiment of the invention does not impose any limitations.

[0095] For example, based on Figure 1 The implementation environment shown in this embodiment of the invention provides a UI design diagram segmentation method based on front-end semantic understanding. The following description uses the application of this UI design diagram segmentation method based on front-end semantic understanding in server 101 as an example. It can be understood that this UI design diagram segmentation method based on front-end semantic understanding can also be applied in terminal 102.

[0096] Reference Figure 2 , Figure 2 This is an optional flowchart of the UI design drawing segmentation method based on front-end semantic understanding provided in the embodiments of the present invention. The execution subject of the UI design drawing segmentation method based on front-end semantic understanding can be any of the aforementioned computer devices (including servers or terminals). Figure 2 The method may include, but is not limited to, steps S100 to S500.

[0097] Step S100: Obtain the UI design drawing to be segmented, preprocess the UI design drawing to obtain the processed design drawing;

[0098] It should be noted that, in some embodiments, preprocessing the UI design image may include at least one of the following steps: scaling the UI design image based on the target size and recording the mapping relationship between the original size and the target size of the UI design image; converting the UI design image to a preset color space; removing image noise from the UI design image using Gaussian filtering; sharpening the blurred areas of the UI design image; and normalizing the grayscale of the UI design image.

[0099] For example, in some specific implementations, the input UI design images (including but not limited to PSD, Sketch, Figma source files, or exported PNG / JPG images) are first standardized. Then, design images of different resolutions are uniformly scaled to a preset size (e.g., 1024×768 pixels), and the mapping relationship between the original size and the normalized size is recorded for subsequent bounding box coordinate restoration. The normalized image is uniformly converted to the RGB color space to eliminate color deviations from export formats of different design tools. Gaussian filtering is then used to remove image noise, and blurred areas are sharpened to ensure clear edges for text and graphic elements. Furthermore, to provide a unified input for subsequent visual feature extraction, grayscale normalization can also be performed on the image.

[0100] Step S200: Based on the processing design diagram, feature extraction is performed using a pre-trained multimodal deep neural network to obtain multimodal features;

[0101] Multimodal features include visual features, text features, and layout features;

[0102] For example, in some specific implementations, multimodal feature hierarchical extraction based on multimodal deep neural networks can be implemented as follows:

[0103] By utilizing pre-trained multimodal deep neural networks (such as a parallel architecture of ViT and OCR), various feature types can be extracted from the design graph, including:

[0104] Visual feature extraction: Extracting the color, texture, shape, etc. of UI buttons and other visual information through a basic visual network framework (such as ViT, CLIP Vision Encoder);

[0105] Text feature extraction: Extract text content, font style, font size, etc. from the UI using text feature extraction methods (such as OCR);

[0106] Layout feature extraction: The coordinate information (x1, y1, x2, y2), aspect ratio, and distance from the canvas edge of each element are recorded by pixel-level scanning and other methods. The hierarchical relationship (z-index value) and alignment mode (horizontal / vertical alignment mark) of the elements are obtained by layer parsing.

[0107] It should be noted that in some embodiments, when the result of feature extraction is visual features, feature extraction using a pre-trained multimodal deep neural network may include the following steps: loading the convolutional layers of the pre-trained visual encoder as the original visual model, freezing the convolutional layers in the original visual model, and initializing to obtain an initial visual model; wherein, the freezing operation includes a first number of convolutional layers before freezing and retaining the last second number of convolutional layers, the sum of the first number and the second number being the total number of convolutional layers in the initial visual model; training the initial visual model using a design map labeled with visual feature tags, constructing an average classification accuracy based on the training feature map output by the penultimate convolutional layer in the initial visual model and the visual feature tags, adjusting the parameters of the initial visual model based on the average classification accuracy to obtain a target visual model; inputting the processing design map into the target visual model, and performing feature dimensionality reduction and encoding on the target feature map output by the penultimate convolutional layer in the target visual model through global average pooling and L2 normalization to obtain visual features.

[0108] For example, in some specific implementations, visual feature extraction can be achieved as follows:

[0109] Feature map extraction is performed based on a pre-trained visual encoder. A pre-trained visual encoder model (such as ResNet-50, EfficientNet-B3, etc.) is selected, the original classification layer of the model is removed, and its convolutional layer is used to extract high-dimensional visual features of the image. The core purpose of "removing the classification layer and retaining the convolutional layer" is that object segmentation requires local visual features, while the global features possessed by the original classification layer cannot serve the task well.

[0110] The specific process is as follows:

[0111] (1) Model loading and initialization: Load the model weights pre-trained on large-scale image datasets such as ImageNet, freeze the first N-2 layers (N is the total number of model layers), retain only the trainability of the last two convolutional layers (or adjust the freezing strategy according to task requirements) to avoid overfitting. Update the parameters of the frozen layers, and the gradient is 0 when freezing.

[0112] (2) Feature map extraction: The normalized webpage screenshot is input into the model, and the feature map output by the penultimate convolutional layer is obtained through forward propagation. The feature map has dimensions (C,H,W) (C is the number of channels, H and W are the spatial dimensions of the feature map), and contains multi-level visual information such as image edges, textures, and semantic regions.

[0113]

[0114] in For the number of semantic categories, For the first True positive, For the first False positives For the first False negative.

[0115] Average classification accuracy (mACC) measures the correctness of region classification. The formula is:

[0116]

[0117] in For the total number of regions, For indicator functions (prediction) With real labels When they are consistent, the value is 1).

[0118] (3) Feature dimensionality reduction and encoding: In order to reduce computational complexity, global average pooling is used on the feature map to compress the (C,H,W) dimension to (C,1,1) to obtain the C-dimensional visual feature vector.

[0119] ,

[0120] in For the feature map aisle, The eigenvalues ​​of the location. Further L2 normalization is used to ensure the eigenvectors satisfy the unity norm constraint, as shown in the formula:

[0121]

[0122] in,

[0123] in For the pooling process 3D eigenvalues Eigenvalues ​​after normalization

[0124] It should be noted that in some embodiments, when the feature extraction result is text features, feature extraction using a pre-trained multimodal deep neural network may include the following steps: performing text recognition on the processing design diagram based on a preset optical character recognition engine to obtain recognized text; locating the bounding box of the text region corresponding to each recognized text through a multi-scale text detection algorithm; recognizing the text content in the bounding box of the text region using a recurrent neural network or a transformer model to obtain structured text; inputting the structured text into a pre-trained large language model, and obtaining text semantic embedding as text features through model forward propagation processing.

[0125] For example, in some specific implementations, text feature extraction can be achieved as follows:

[0126] OCR (Optical Character Recognition) + Semantic Embedding of a Large Language Model: For text information in webpage screenshots, the process combines OCR recognition with semantic encoding using a large language model, as follows:

[0127] (1) OCR text recognition: Industrial-grade OCR engines (such as Tesseract OCR, PaddleOCR, etc.) are used to detect and recognize text in web page screenshots. Multi-scale text detection algorithms (such as EAST, DBnet) are used to locate the bounding boxes of text regions, and then recurrent neural networks (RNN) or Transformer models are used to recognize the text content and output structured text (such as key texts such as "AIflow" and "domestic version").

[0128] In OCR and text detection tasks, text detection recall, character accuracy (CER), and processing time (ms) are key metrics for evaluating model performance, and their calculation formulas are as follows:

[0129] 1. Text Detection Recall:

[0130] Text detection recall measures the proportion of real text regions successfully detected by a model out of all real text regions, reflecting the model's ability to "not miss any detections". Its calculation formula is as follows:

[0131]

[0132] Correctly detected text regions typically refer to those where the intersection-over-union (IoU) between the bounding boxes detected by the model and the bounding boxes of the actual text regions exceeds a preset threshold (e.g., 0.5). For example, if there are actually 10 text regions in an image, and the model successfully detects 8 (IoU meets the threshold), then the recall rate is [missing information]. .

[0133] 2. Character Error Rate (CER):

[0134] Character accuracy (CER) measures the error rate at the character level in OCR recognition results, reflecting the precision of text recognition (the lower the error rate, the higher the accuracy). Its calculation formula is:

[0135]

[0136] Edit distance (Levenshtein distance) refers to the minimum number of single-character operations required to convert the recognition result into real text. This includes insertion (adding a character), deletion (deleting a character), and substitution (replacing one character with another). For example, if the real text is "AI technology" and the recognition result is "AI technique description", the edit distance is 1 (substituting "technique" → "description"). .

[0137] 3. Processing Time (ms):

[0138] Processing time refers to the total time consumed by the model from input image (such as webpage screenshot) to output final recognition result, measured in milliseconds (ms), reflecting the model's efficiency.

[0139] Processing time = End time (output result time) - Start time (input image time);

[0140] Multiple tests are typically conducted, and the average value is taken to reduce random errors (such as the impact of hardware load fluctuations). If a model takes 5000ms to process 100 images, then the average processing time is... .

[0141] These metrics comprehensively evaluate the performance of an OCR system from three dimensions: "detection integrity," "recognition accuracy," and "processing efficiency," and serve as an important basis for model optimization and comparison.

[0142] Table 1 below shows the test results of these three metrics for three types of web pages using the Tesseract and PaddleOCR engines:

[0143] Table 1

[0144]

[0145] The specific output results corresponding to different outputs are as follows:

[0146] Example 1:

[0147] Input screenshot: News search results page (including title, summary, and timestamp);

[0148] ORC output: [

[0150] {"text":"Frontiers of Science and Technology","bbox":[100,50,300,80],"confidence":0.98},

[0151] {"text":"AI Helps Achieve Scientific Breakthroughs..............","bbox":[120,120,700,160],"confidence":0.97},

[0152] {"text":"Read the full article","bbox":[650,250,750,270],"confidence":0.95},

[0153] {"text":"3 weeks ago","bbox":[700,170,800,190],"confidence":0.93} ]

[0155] Example 2:

[0156] Input screenshot: Paper search results page (including title, author, conference information);

[0157] OCR output: [

[0159] {"text":"Research on Medical Image Segmentation Algorithm Based on Deep Learning","bbox":[150,100,700,140],"confidence":0.96},

[0160] {"text":"An improved U-Net model is proposed, which combines an attention mechanism to achieve high-precision segmentation of lung CT images...","bbox":[150,150,500,170],"confidence":0.94},

[0161] {"text":"Computer Applications Citations: 128 (March 15, 2024)","bbox":[150,180,600,200],"confidence":0.92}, ]

[0163] (2) Text semantic embedding generation: The identified text is input into a pre-trained large language model (such as BERT, ERNIE, etc.), and the text semantic embedding is obtained through forward propagation of the model.

[0164] 2.1 First, the text is preprocessed: the OCR-recognized text is cleaned, invalid characters (such as garbled characters and repeated spaces) are filtered out, and word segmentation and encoding (such as BERT's WordPiece segmentation) are performed according to the model input requirements to generate a fixed-length input sequence.

[0165] The following is an example of the processing result:

[0166] 1) News search page text processing:

[0167] OCR original output: "AI helps scientific research break through, this technology is changing the future\nRead full text";

[0168] Cleaned text: "AI helps scientific research break through, this technology is changing the future Read full text";

[0169] WordPiece tokenization result: ["[CLS]","ai","helps","scientific","research","break","through",",","this","technology","is","changing","the","future","read","full","text","[SEP]"]

[0170] Encoded input sequence:

[0171] [101,2009,687,139,4692,5609,4083,1384,8024,6821,1767,2207,3698,6206,4415,2192,5385,7227,6851,6850,6356,1394,6855,102,0,0,0,0,0,0,0];

[0172] 2) Academic search page text processing:

[0173] OCR original output: "Research on medical image segmentation algorithm based on deep learning\nPropose an improved U-Net model, combine attention mechanism to achieve high-precision segmentation of lung CT images...\n2024-03-15 Cited by Computer Applications: 128";

[0174] Cleaned text: "Research on medical image segmentation algorithm based on deep learning Propose an improved U-Net model, combine attention mechanism to achieve high-precision segmentation of lung CT images...2024-03-15 Cited by Computer Applications: 128";

[0175] WordPiece segmentation results: ["[CLS]","Based on","Deep","Learning","Learning","Of","Medical","Learning","Image","Segmentation","Calculation","Method","Research","Propose","Out","One","Kind","Improvement","Of","u","-","net","Model","Type",",","Combined","Attention","Force","Mechanism","System","Realization","To","Lung","Part","ct","Image","Of","High","Precision","Degree","Segmentation","...","2024","-","03","-","15","Calculation","Computer","Application","Application","Be","Introduced",":","128","[SEP]"];

[0176] Encoded input sequence:

[0177] [101,3315,1744,3570,1920,2110,1282,4638,3019,2110,5011,5632,2399,3698,3215,3185,3329,2774,3189,1303,1744,3321,6209,3297,4638,1053,1011,17924,3079,2265,1010,3221,2636 ,6840,6741,2339,3214,3196,3301,1303,1971,6637,6381,1059,5011,5632,4638,3563,7604,1920,2399,3698,8043,1013,107,108,109,6382,3152,2825,3490,3018,511,128,102,0,0,...,0];

[0178] 2.2. Semantic encoding is then performed: the encoded text sequence is input into a large language model, and the hidden layer output corresponding to the [CLS] token is extracted as the text semantic embedding vector. This vector typically has a dimension of 768 (BERT-base model) or higher, and can represent the contextual semantic information of the text.

[0179] The code flow for semantic encoding using the BERT model as an example:

[0180] import torch

[0181] from transformers import BertModel, BertTokenizer

[0182] # Load the pre-trained model and word segmenter

[0183] tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

[0184] model = BertModel.from_pretrained('bert-base-uncased')

[0185] # Example Input (Text from E-commerce Search Page)

[0186] text = "Limited-time special offer: New mobile phones up to 50% off! Buy now!"

[0187] inputs = tokenizer(text, return_tensors="pt", padding="max_length",max_length=32)

[0188] # Model forward propagation

[0189] with torch.no_grad():

[0190] outputs = model(**inputs)

[0191] # Extract the embedding vector of the [CLS] token (dimensions: [1, 768])

[0192] cls_embedding = outputs.last_hidden_state[:, 0, :]# Shape: [batch_size,768]

[0193] # L2 Normalization

[0194] normalized_embedding = torch.nn.functional.normalize(cls_embedding, p=2, dim=1)

[0195] L2 normalization is performed on the semantic embedding vectors, using the same formula as for visual feature normalization, to ensure that different text embeddings have a unified metric.

[0196] It should be noted that in some embodiments, when the result of feature extraction is layout features, feature extraction using a pre-trained multimodal deep neural network may include the following steps:

[0197] Once the HTML source code of the UI design draft is obtained, layout features based on the DOM structure are extracted. This DOM-based layout feature extraction includes the following operations: constructing a DOM tree structure based on the HTML source code; the DOM tree structure includes the element hierarchy of all DOM elements; converting the CSS style information in the HTML source code to obtain the bounding box coordinates of each DOM element; matching and identifying key elements of the DOM elements through tag names and / or class names, and generating layout features based on these key elements; the layout features include the bounding box coordinates of the key elements, as well as the hierarchical relationships and the number of sibling elements determined based on the element hierarchy.

[0198] When the HTML source code of the UI design is not available, layout feature inference based on visual analysis is performed on the processed design. This visual feature inference includes the following operations: dividing the processed design into text and non-text regions using optical character recognition combined with image segmentation algorithms; treating each text region as a text element and outputting the bounding box coordinates of the text region based on the optical character recognition results; performing contour detection on the non-text regions, and then strengthening weak edges through adaptive thresholding and edge enhancement to obtain non-text elements and their corresponding bounding box coordinates; converting the top-bottom and overlapping relationships of text and non-text elements in the processed design to obtain element hierarchy; and generating layout features based on the bounding box coordinates and element hierarchy.

[0199] For example, in some specific implementations, layout feature extraction can be achieved as follows:

[0200] Structural analysis layout features, combining DOM parsing and visual inference, are used to describe the spatial relationships and hierarchical structure of web page elements. This is achieved through two methods: DOM structure parsing (if HTML source code is available) or visual analysis (relying solely on screenshots).

[0201] (1) Layout feature extraction based on DOM structure (scenario with HTML source code):

[0202] For example, the HTML structure of the header area (including logo, navigation bar, and search box) on a news website homepage is shown below:

[0203] <header class="site-header">

[0204]

[0205]

[0206]

[0207] <nav class="main-nav" style="margin-left:200px; top:30px;">

[0208]

[0209] front page

[0210] news

[0211] video

[0212]

[0213] < / nav>

[0214]

[0215] <input type="text" placeholder="搜索新闻">

[0216]

[0217] < / header>

[0218] First, parse the webpage's HTML code to construct a DOM tree structure. Using tools such as BeautifulSoup and PyDOM, traverse the HTML tags to generate a tree structure containing element type (tagName), attributes (className, id), and parent-child relationships.

[0219] Spanning tree structure:

[0220] header (root node) → div.logo, nav.main-nav, div.search-box (child node);

[0221] nav.main-nav→ul→li*3 (grandchild nodes).

[0222] Then, combining CSS style information (such as offsetTop, offsetLeft, width, and height), the bounding box coordinates (x1, y1, x2, y2) of each DOM element on the page are calculated ((x1, y1) are the top-left corner coordinates, and (x2, y2) are the bottom-right corner coordinates). The element hierarchy (such as navigation bars) is determined through the nesting relationships of the DOM tree. <nav>Top-level search results list (For sub-levels).

[0223] Logo area: (20, 10, 140, 70) (left=20px, top=10px, width=120px, height=60px).

[0224] Navigation bar `li` items: First `li` coordinates (200, 30, 250, 50) (margin-left=200px, top=30px, padding=0.15px);

[0225] Search box: (774, 30, 1074, 60) (Page width 1024px, right=50px → left=1024-50-300=674px).

[0226] Key element identification and encoding: Define key layout elements (such as search boxes, navigation bars, buttons, etc.) and identify them by matching tag names (tagName) and class names (className). For each key element, encode its bounding box coordinates, hierarchical relationship, number of sibling elements, and other features to generate a layout feature vector.

[0227]

[0228] (2) Layout feature inference based on visual analysis (scenario without HTML source code) When the HTML source code cannot be obtained, the layout structure is inferred through computer vision algorithms:

[0229] Text and Non-Text Region Segmentation: Utilizing OCR recognition results and combining them with image segmentation algorithms (such as U-Net), text regions and non-text regions (such as buttons, icons, and backgrounds) in webpage screenshots are distinguished. For text regions, the bounding boxes output by OCR are used directly; for non-text regions, bounding boxes are extracted through contour detection (OpenCV's findContours). When non-text regions lack clear contours (such as gradient background areas, transparent elements, low-contrast areas, etc.), relying solely on OpenCV's findContours for contour detection will fail. In this case, multimodal feature fusion and contextual analysis are needed to assist in recognition. For example, weak edges can be strengthened through preprocessing such as adaptive thresholding and edge enhancement before attempting contour detection again. The layer hierarchy (such as upper-layer elements occluding lower-layer elements) is inferred by utilizing the vertical and overlapping relationships of elements in the screenshot.

[0230] like Figure 3 As shown, taking a screenshot of an academic literature retrieval platform webpage as an example, OCR is used to recognize text regions, such as paper titles (e.g., "Research on Medical Image Segmentation Algorithm Based on Deep Learning"), publication information ("Computer Applications, cited 128 times on 2024-03-15"), search box prompt text ("Search academic papers, authors..."), etc., and an example of outputting bounding boxes is shown below.

[0231] {"Research on Medical Image Segmentation Algorithm Based on Deep Learning": (130, 380, 390, 400)}

[0232] "Computer Applications" cited 128 times on March 15, 2024: (130, 410, 450, 430).

[0233] "Search academic papers, authors...": (170, 150, 690, 180)}(80, 150, 320, 350), Button icon outline: (350, 230, 450, 260).

[0234] Non-text areas, such as the outlines of the paper list cards and the search button icon, are bounding boxes extracted using OpenCV.findContours: the outline of the first paper card in the paper list is (120, 350, 700, 500), and the outline of the search button icon is (720, 145, 820, 190).

[0235] like Figure 4 As shown, taking a screenshot of a news platform webpage as an example, OCR is used to recognize text regions, such as news titles (e.g., "AI helps scientific breakthroughs and accelerates drug development"), published information ("Science and Technology Daily, 23,000 reads, July 22, 2025"), search box suggestion text ("Search news..."), etc., and an example of outputting bounding boxes is given.

[0236] "AI Helps Achieve Scientific Breakthroughs and Accelerate Drug Development": (130, 380, 370, 400)

[0237] "Science and Technology Daily, July 22, 2025, 23,000 views": (130, 410, 420, 430)

[0238] "Search news...": (170, 150, 690, 180)}

[0239] Non-text areas, such as the outlines of news list cards and search button icons, are extracted using OpenCV.findContours: the outline of the first news list card is (120, 350, 700, 500), and the outline of the search button icon is (720, 145, 820, 190).

[0240] Key element semantic mapping:

[0241] By combining visual features and textual semantics, semantic classification is performed on bounding box regions (e.g., identifying button icons using visual models such as CNNs and determining them as "search buttons" based on nearby text), supplementing the semantic information of layout features. Through the above preprocessing and feature extraction process, visual features (image content representation), textual features (semantic information), and layout features (structural relationships) can be obtained from webpage screenshots, providing multi-dimensional input for subsequent tasks such as webpage similarity assessment and redundancy detection. This ensures that features cover the visual, semantic, and structural attributes of the webpage, improving the robustness and accuracy of the evaluation model.

[0242] Example of an academic search scenario:

[0243] Based on the positional relationship: Paper list card (bottom layer, containing text areas such as paper title and publication information) → Search box (covering the top of the page, not obstructed by the paper list) → Search button (to the right of the search box, not obstructed), the inferred hierarchy is: Paper list card (level 1) < Search box (level 2) = Search button (level 2).

[0244] Visual features of the search button area (rectangle, blue background, etc.) + text "Search" (suggested text in the nearby search box) → identified as "search function button"; Academic paper card aggregation: merge the bounding boxes of paper title, publication information, and card outline, such as merging the first paper into (120, 350, 700, 500) and marking it as "academic document card".

[0245] Example of a news search scenario:

[0246] Based on the positional relationship: News list card (bottom layer, containing text areas such as news titles and published information) → Search box (overlaying the top of the page, not obstructed by the news list) → Search button (to the right of the search box, not obstructed), the inferred hierarchy is: News list card (level 1) < Search box (level 2) = Search button (level 2).

[0247] Search button area visual features (rectangle, blue background, etc.) + text "Search" (nearby search box prompt text) → identified as "search function button"; News information card aggregation: merge the bounding boxes of news titles, published information, and the outline of the card, such as merging the first news into (120, 350, 700, 500), and marking it as "news information card".

[0248] Step S300: Based on multimodal features, perform component semantic segmentation on the processing design diagram to obtain component data;

[0249] The component data includes the original boundaries and component categories of each atomic-level component in the processing design diagram;

[0250] It should be noted that in some embodiments, such as Figure 5 As shown, step S300 may include the following steps: S310, using a semantic segmentation network to process and obtain the atomic-level components and their corresponding original boundaries in the processing design diagram; S320, using a lightweight classifier to determine the semantic labels of the detection regions corresponding to each original boundary to obtain the component category of each atomic-level component.

[0251] For example, in some specific implementations, UI element detection and semantic type classification can be achieved as follows: Based on the extracted multimodal features, a semantic segmentation network (such as the DeepLab series or Encoder-Decoder architecture) is used to locate and segment atomic components (buttons, icons, text boxes, etc.) in the UI image. Simultaneously, a lightweight classifier (such as CLIP TextEncoder) is used to determine the semantic label of each detected region, thus identifying its component category.

[0252] Step S400: Based on the original boundaries and component categories, perform rule-based clustering on atomic-level components to construct a multi-level semantic hierarchy system;

[0253] It should be noted that in some embodiments, step S400 may include the following steps: using atomic-level components as fine-grained regions at the component level, using the original boundaries as component-level bounding boxes, and using component categories as component-level semantic labels; determining the spatial relationships between each atomic-level component based on the original boundaries, and then performing component clustering based on component categories to obtain module-level medium-grained regions; marking module-level bounding boxes based on the region range corresponding to the medium-grained regions, and performing semantic label determination on the detection regions corresponding to the module-level bounding boxes to obtain module-level semantic labels for each medium-grained region; using rule templates to divide the processing design diagram into multiple large regions based on page layout logic, and classifying each medium-grained region into the corresponding large region to obtain page-level coarse-grained regions; marking page-level bounding boxes based on the region range corresponding to the coarse-grained regions, and performing semantic label determination on the detection regions corresponding to the page-level bounding boxes to obtain page-level semantic labels for each coarse-grained region.

[0254] For example, in some specific implementations, region-level structure aggregation and hierarchical semantic generation can be achieved as follows:

[0255] A three-level semantic hierarchy is constructed based on the aggregation results:

[0256] Fine-grained areas (component level): Preserve the original boundaries of individual semantic elements, such as "password input box" and "confirm button", as the most basic interactive units.

[0257] Medium-grained areas (module level): These consist of fine-grained areas that are related to functions, such as the "Login Form Module" (which includes a username input box, a password input box, and a login button) and the "Product Card Module" (which includes product images, name text, and price tags).

[0258] Coarse-grained areas (page level): Large areas divided according to page layout logic, such as "top navigation area", "main content area" and "bottom copyright area", each containing multiple medium-grained modules.

[0259] Low-level atomic components are aggregated into mid-level functional modules (such as search bars, list modules, and navigation bars) based on spatial and semantic adjacency relationships, and then further constructed into high-level page structures (such as headers, body, and footers). This process combines rule templates with clustering algorithms (such as MeanShift or DBSCAN) to complete the module hierarchy division.

[0260] Step S500: Output the UI design drawing segmentation result according to the multi-level semantic hierarchy system;

[0261] For example, in some specific implementations, corresponding bounding boxes and semantic labels are generated for all segmented regions, and the semantic segmentation results in the UI image are output in a structured format (such as JSON / XML or graph database representation). This structured result can be directly used for subsequent UI reconstruction, code generation, or usability analysis tasks.

[0262] To explain in detail the principle of the technical solution of the present invention, the overall process of the present invention will be described below with reference to some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and should not be regarded as a limitation of the present invention.

[0263] First, it's important to note that with the rapid development of the internet and mobile applications, user interface (UI) design plays an increasingly crucial role in software development. UI design drafts serve as a bridge between product design and development; their accurate understanding and efficient processing are essential for accelerating the development process and improving user experience. Traditional UI design draft processing and front-end development workflows typically rely on manual analysis and coding, which is not only inefficient but also prone to introducing human error. To improve efficiency, the industry has developed various automated UI design draft processing technologies, among which UI design draft segmentation is a key component.

[0264] Existing UI design image segmentation methods primarily focus on visual feature-based segmentation, such as using image processing techniques (e.g., edge detection, color clustering, texture analysis) or deep learning-based computer vision models (e.g., object detection, instance segmentation) to identify and divide visible elements or regions in the UI. These methods can effectively physically separate different visual components (e.g., buttons, text boxes, images) or layout areas (e.g., header, body, footer) in UI design images. For example, some methods train convolutional neural networks (CNNs) to identify UI elements and generate their bounding boxes or pixel-level masks, thereby achieving automated extraction of UI elements. Other methods may focus on identifying the overall layout structure of the UI, dividing the page into several rectangular regions, each corresponding to a logical module.

[0265] In addition, some studies have attempted to segment UI elements by incorporating spatial relationships between them. For example, graph neural networks (GNNs) are used to model the connections and hierarchical relationships between UI elements, thereby assisting in region division. These methods have improved the automation capabilities of UI design drafts to some extent, laying the foundation for subsequent applications such as code generation and UI automation testing.

[0266] While existing UI design draft segmentation methods have made some progress in visual recognition and segmentation, they generally suffer from limitations and technical problems that restrict their effectiveness and intelligence in practical applications. In view of these shortcomings, this invention aims to address the problems in existing UI design draft segmentation methods, such as a lack of deep semantic understanding, poor adaptability to complex UIs, difficulty in handling non-explicit semantic information, and a mismatch between segmentation granularity and semantic meaning. By introducing the concept of front-end semantic understanding, this invention proposes a novel UI design draft segmentation method that combines visual and semantic information to achieve accurate and intelligent segmentation of different semantic regions in UI design drafts, thereby providing a foundation for higher-level UI automation and intelligent development.

[0267] This invention aims to address the technical problems existing in UI design diagram segmentation methods, such as insufficient semantic understanding, poor adaptability to complex UIs, difficulty in handling non-explicit semantic information, and mismatch between segmentation granularity and semantics. It proposes a UI design diagram segmentation method based on front-end semantic understanding. Specific objectives of this invention include:

[0268] (1) Achieve deep semantic segmentation of UI design drawings: Break through the limitations of traditional segmentation based on visual features, and achieve semantic understanding and precise division of each area in the UI design drawing by combining visual information, text content, layout structure and potential interaction logic. For example, it can identify semantic areas with clear functions and meanings, such as "navigation bar", "search area", "product display area" and "user comment area", rather than just physical rectangles.

[0269] (2) Enhanced robustness to complex and diverse UIs: For UI design drafts with different design styles, complex layouts, and custom components, this invention can maintain high segmentation accuracy and generalization ability. By introducing semantic understanding, the model can better adapt to the diversity of UI designs and reduce dependence on specific visual patterns.

[0270] (3) Effectively utilize non-explicit semantic information: Make full use of non-visual semantic clues in UI design drawings, such as text labels, placeholders, icon meanings, and logical relationships between elements, to assist in more accurate area division and semantic recognition. For example, distinguish different types of input boxes or buttons by analyzing text content.

[0271] (4) Supports multi-granularity, semantic matching segmentation results: It can provide segmentation results of different granularities according to needs, and ensure that the segmented regions are highly matched with the actual semantic structure of the UI. For example, it can not only identify the entire "news list" area, but also further identify the semantic region of each "news item" and understand its internal elements such as title, summary, and image.

[0272] (5) Provide a foundation for UI automation and intelligent development: By providing high-precision and highly intelligent semantic segmentation results, it provides more reliable and intelligent input for subsequent applications such as UI automated testing, intelligent front-end code generation, UI component library construction, and accessibility assistance tool development, thereby greatly improving development efficiency and product quality.

[0273] like Figure 6 As shown, this invention provides a UI design drawing segmentation method based on front-end semantic understanding. Its core lies in combining multimodal information (visual, textual, and layout) and deep learning technology to achieve accurate identification and segmentation of different semantic regions within the UI design drawing. The method flow of this invention is as follows: Figure 6 As shown, the main steps include the following:

[0274] (1) UI design input and preprocessing: Receive original UI design images (such as webpage screenshots, mobile application interface images, etc.) as input. Perform necessary preprocessing on the images, such as size normalization and color space conversion, to adapt to the input requirements of subsequent models.

[0275] (2) Multimodal Feature Extraction: Utilize deep learning models such as Convolutional Neural Networks (CNNs) to extract rich visual features from UI design drawings, capturing information such as the shape, color, texture, and style of elements. Combine this with Optical Character Recognition (OCR) technology to identify and extract all readable text content from the UI design drawings. Encode the extracted text, for example, using word embeddings or a Transformer encoder to obtain its semantic information. Analyze the layout information of elements in the UI design drawings, such as their spatial position, size, hierarchical relationship, and alignment. This can be achieved by constructing an adjacency graph, hierarchical tree, or using a dedicated layout encoder.

[0276] (3) Semantic element recognition and relationship construction: Using object detection or instance segmentation models, various basic UI elements (such as buttons, input boxes, images, icons, text blocks, etc.) in the UI design diagram are identified, and their bounding boxes and category information are obtained. Based on the identified basic UI elements, spatial relationships (such as adjacent, contained, overlapping) and potential logical relationships (such as belonging to the same component or constituting the same functional module) between them are constructed. This can be modeled using graph neural networks (GNN) or attention mechanisms.

[0277] (4) Contextual Semantic Analysis and Region Aggregation: Visual features, text features, layout features, and element relationship information are fused and input into a multimodal fusion model. This model can understand the correlation between different modal information and capture the contextual semantics of the UI from both global and local levels. Based on the fused contextual information, the model identifies regions with specific semantic functions through inference mechanisms. For example, by analyzing the text "search" and the adjacent input boxes and buttons, it infers that this is a "search region". Basic UI elements and sub-regions with the same or related semantics are aggregated into larger semantic regions. For example, multiple product images, names, prices, and purchase buttons are aggregated into a "product card" region, and then multiple product cards are aggregated into a "product list" region. This process can be implemented using hierarchical clustering, graph segmentation algorithms, or Transformer-based sequence generation models.

[0278] (5) Generation and Output of Multi-Granularity Semantic Regions: This invention can output semantic regions of different granularities according to requirements. For example, it can output coarse-grained "navigation bar", "main content area", and "footer", as well as medium-grained "search box" and "user avatar area", and even fine-grained "submit button" and "username input box". Finally, it outputs the bounding box coordinates, corresponding semantic labels (such as "navigation bar", "search area", "product list", etc.) and confidence scores of each identified semantic region. These results can be represented in JSON, XML or other structured data formats for easy use in subsequent applications.

[0279] In some preferred embodiments, a structured segmentation result dataset can be generated, which can be specifically implemented as follows:

[0280] Semantic tagging system: assign multi-level tags to each area, such as "Main content area > Product list module > Product images", and the tag naming follows the W3C UI component specification;

[0281] Bounding box parameters: Record the region coordinates in the format of (x_min, y_min, x_max, y_max), and output the relative coordinates (percentage of the width and height of the design drawing) and absolute pixel coordinates;

[0282] Confidence score: A confidence value of 0-1 is added to each region, and the score is calculated by combining indicators such as element recognition accuracy and cluster stability.

[0283] The final output format supports JSON, COCO annotation format, and visual annotation charts. The visual results use different colored borders to distinguish granularity levels (red - fine-grained, blue - medium-grained, green - coarse-grained), with core semantic tags annotated next to the borders. Core semantic annotations consist of multiple layers, as follows: Fine-grained area: Core semantic tags correspond to "component-level" tags (the lowest level in the three-level semantic hierarchy), which are the semantic names of individual atomic components, such as "password input box," "confirm button," and "product image." This directly describes the core function / identity of the fine-grained area, excluding higher-level tags (such as "login form module" and "main content area"). Medium-grained area: Core semantic tags correspond to "module-level" tags (the middle layer of the three-level semantic hierarchy), which are the names of modules aggregated from functionally related fine-grained components, such as "login form module," "product card module," and "search bar." This only reflects the core function of the medium-grained module, excluding lower-level component tags (such as "username input box") and higher-level area tags (such as "main content area"). Coarse-grained area: The core semantic tag corresponds to the "page-level" tag (the top level of the three-level semantic hierarchy), which is the name of a large area divided according to the page layout logic, such as "top navigation area", "main content area" and "bottom copyright area". It only describes the overall positioning of the coarse-grained area and excludes lower-level module tags (such as "product list module").

[0284] In some specific application scenarios, taking semantic segmentation of web search results pages as an example, this embodiment of the invention implements semantic segmentation of web page UI design diagrams to achieve automatic identification of different functional areas of web pages. Two types of typical web page screenshots are selected, such as... Figure 3 and Figure 4 As shown in the examples (academic search page and news search page), the original sizes were 1600×900 and 1920×1080 respectively, and they were uniformly scaled to 1024×768. This can be achieved through the following process:

[0285] Step 1: Input and Image Preprocessing:

[0286] To eliminate the interference of screenshot size differences caused by different devices and rendering environments on feature extraction, webpage screenshots are uniformly scaled to a preset standard size (e.g., 1024×768 pixels). The scaling process uses a bilinear interpolation algorithm to maintain the proportions of key image structures while avoiding excessive pixel distortion. Size conversion is implemented using image processing libraries such as OpenCV, with the following formula:

[0287] Let the original screenshot size be... The target size is Then the scaled coordinates Compared with the original coordinates ( )satisfy:

[0288]

[0289] After scaling, the image is grayscale normalized, mapping pixel values ​​to the [0, 1] range to provide a unified input for subsequent visual feature extraction.

[0290]

[0291] in These are the original pixel values. , These are the global minimum and maximum pixel values ​​of the image.

[0292] Step 2: Feature extraction. The specific principles, logic, and examples are explained in the previous embodiments and will not be repeated here.

[0293] Step 3: Contextual semantic analysis and region aggregation:

[0294] (1) Fusion and Reasoning:

[0295] The model integrates visual, textual, layout, and relational information. Through inference (with an image as input), it identifies the area at the top containing the search box, logo, and navigation options as the "header area / search area." It identifies the area below containing the search results list as the "main content area / search results list." The "Others also searched" area on the right is identified as the "sidebar / recommendations area."

[0296] (2) Fine-grained polymerization:

[0297] Within the "Main Content Area," the model further aggregates each independent search result entry into a "Search Item" semantic area, and each search result item contains sub-areas such as "Title," "URL," and "Summary." Within the "Sidebar / Recommendation Area," each recommended search term is identified as a "Recommended Search Item" semantic area.

[0298] Example 1, Regional aggregation on a technology search page:

[0299] (1) Multi-feature fusion in reasoning: Visual features show that the middle of the page is a search results list (including text blocks such as title, abstract, and number of views). Text features identify news titles such as "AI helps scientific research breakthroughs" and "New breakthroughs in quantum computing" as well as information such as number of views and source. Layout features show that this area occupies the main width of the page (level 2, with a large horizontal proportion). The fusion inference is "search results list area".

[0300] Key area reasoning:

[0301] Header area: includes navigation (Home, News, Videos, More) and search box, bounding box (0, 0, 1024, 100);

[0302] Main content area: (0, 100, 750, 920) (containing 3 technology news search results);

[0303] Sidebar area: (750, 100, 1024, 920) (Text "Hot Recommendations" and content such as AI drug development and quantum computing progress, arranged on the right).

[0304] (2) Fine-grained polymerization:

[0305] Based on text semantics (relationships between information such as title, source, and readership) and layout (clear spacing between news items), the aggregation is as follows:

[0306] Search result item 1 (Technology News 1): (120, 150, 720, 320), sub-area:

[0307] Title: (130, 150, 400, 180) (Text "AI Helps Scientific Breakthroughs and Accelerates Drug Development")

[0308] Abstract area: (130, 190, 720, 280) (Text "A research team, with the help of artificial intelligence technology, has reduced the traditional drug screening time to several days...").

[0309] Additional information: (130, 290, 350, 310) (text "2025-07-22 Science and Technology Daily, 23,000 reads").

[0310] Search result item 2 (Technology News 2): (120, 350, 720, 520), sub-area:

[0311] Title: (130, 350, 400, 380) (Text "New Breakthrough in Quantum Computing: Achieving Entanglement of 50 Quantum Bits")

[0312] Abstract area: (130, 390, 720, 480) (Text "International research team successfully maintains stable entanglement of 50 qubits for more than 100 seconds, breaking the record...").

[0313] Additional information: (130, 490, 350, 510) (Text "2025-07-21 Scientific American" 16,000 reads).

[0314] Search result item 3 (Technology News 3): (120, 550, 720, 720), sub-region:

[0315] Title: (130, 550, 400, 580) (Text: "New Energy Vehicle Sales Hit Record High, Market Landscape Shifts");

[0316] Abstract area: (130, 590, 720, 680) (Text "New energy vehicle sales data for the first half of the year released, industry penetration rate exceeds 30% for the first time...");

[0317] Additional information: (130, 690, 350, 710) (Text "2025-07-20 Caijing.com 32,000 reads").

[0318] Example 2, Regional aggregation on the academic search page:

[0319] (1) Multi-feature fusion in reasoning: Visual features show that the middle area is the search results list (including text blocks such as paper title, abstract, journal, etc.), text features identify academic literature related terms such as "research on medical image segmentation algorithm based on deep learning" and "cited", layout features show that this area occupies the main width of the page (level 2, with a large horizontal proportion), and the fusion inference is "academic literature search results list area".

[0320] Key area reasoning:

[0321] Header area: includes navigation (Home, Papers, Scholars, Journals, More) and search box, bounding box (0, 0, 1024, 100);

[0322] Main content area: (0, 100, 750, 920) (contains 3 academic literature search results);

[0323] Sidebar area: (750, 100, 1024, 920) (Text "Popular Research Areas" and content such as medical imaging AI and bioinformatics are arranged on the right).

[0324] (2) Fine-grained polymerization:

[0325] Based on textual semantics (relevance of document information such as title, journal, and citation count) and layout (clear spacing between document items), the aggregation is as follows:

[0326] Search result item 1 (Academic Literature 1): (120, 150, 720, 320), sub-region:

[0327] Title: (130, 150, 400, 180) (Text "Research on Medical Image Segmentation Algorithm Based on Deep Learning");

[0328] Abstract region: (130, 190, 720, 280) (Text "Proposes an improved U-Net model, combined with an attention mechanism to achieve high-precision segmentation of lung CT images...").

[0329] Additional information: (130, 290, 350, 310) (text "2024-03-15 Computer Applications cited: 128").

[0330] Search result item 2 (Academic Literature 2): (120, 350, 720, 520), sub-region:

[0331] Title: (130, 350, 400, 380) (Text "Design and Implementation of an Artificial Intelligence-Assisted Clinical Decision-Making System")

[0332] Abstract area: (130, 390, 720, 480) (Text "Design of a multimodal fusion-based clinical decision support system integrating electronic medical records and medical imaging data...").

[0333] Additional information: (130, 490, 350, 510) (text "Cited by Chinese Medical Journal on 2023-11-22: 09").

[0334] Search result item 3 (Academic Literature 3): (120, 550, 720, 720), sub-region:

[0335] Title: (130, 550, 400, 580) (Text "Application Research of Federated Learning in Medical Data Privacy Protection")

[0336] Abstract area: (130, 590, 720, 680) (Text "Addressing the issue of medical data privacy, a distributed model training framework based on federated learning is proposed...").

[0337] Additional information: (130, 690, 350, 710) (text "Journal of Software cited 95 times on 2024-01-08").

[0338] Step 4: Output the final semantic segmentation result, which will include:

[0339] Header / Search Region: Bounding box coordinates, semantic labels.

[0340] Main content area / search results list: bounding box coordinates, semantic tags.

[0341] Sub-region: Search result item 1 (bounding box, semantic label), which includes sub-regions such as title, URL, and summary.

[0342] Sub-region: Search result item 2 (bounding box, semantic label), and so on.

[0343] Sidebar / Recommended Area: Bounding box coordinates, semantic tags.

[0344] Sub-region: Recommended search term 1 (bounding box, semantic label).

[0345] Sub-region: Recommended search term 2 (bounding box, semantic label), and so on.

[0346] The detailed process is supplemented as follows:

[0347] 1. First, determine the core elements of the visualization: use different colored borders to mark areas (e.g., red borders mark the inferred title area, blue borders mark the navigation area), and add semantic labels next to the borders (e.g., "no outline - title area");

[0348] 2. Based on the output region coordinates (x_min, y_min, x_max, y_max), use tools such as OpenCV / PyQt to draw the corresponding borders on the original UI image. For areas without outlines, add semi-transparent fills (such as light red fills) to enhance recognizability.

[0349] 3. Finally, overlay a legend (explaining the meaning of colors / labels) to generate a labeled visualization, which can also support interaction (such as displaying area details by hovering the mouse).

[0350] The following table 2 shows an example of the output results from the academic search page:

[0351] Table 2

[0352]

[0353] The simplified visualization of the result is shown in the semantic segmentation graph. Figure 7 As shown.

[0354] The following table 3 shows an example of the output results for the news search page:

[0355] Table 3

[0356]

[0357] The output results are simply visualized as follows: Figure 8 As shown.

[0358] In summary, the present invention, through the above technical solution, can effectively solve the pain points of UI design drawing segmentation in the prior art, and brings significant technical effects:

[0359] (1) Solve the problem of lack of deep semantic understanding: It can extract deep semantics from visual and non-visual information to achieve accurate identification of the functions and meanings of UI areas, rather than just physical segmentation.

[0360] (2) Improve adaptability to complex UIs: By combining multimodal features and semantic reasoning, the model has stronger generalization ability and robustness to UI design diagrams with different design styles and complex layouts.

[0361] (3) Effective use of non-explicit semantic information: It can automatically analyze non-explicit information such as text content and icon meaning, so as to more accurately identify the true semantics of UI elements.

[0362] (4) Provides multi-granularity segmentation results for semantic matching: It can output semantic regions of different granularities according to application requirements, and these regions are highly consistent with the actual logical structure of the UI, which greatly improves the usability of the segmentation results.

[0363] (5) Significantly improve UI automation and intelligent development efficiency: Accurate semantic segmentation results provide high-quality input for subsequent automation tasks, reduce manual intervention, accelerate the development cycle, reduce error rate, and thus improve overall development efficiency and product quality.

[0364] Compared with the prior art, the key technical points and corresponding beneficial effects achieved by the embodiments of the present invention are mainly reflected in the following aspects:

[0365] 1. UI semantic understanding through multimodal feature fusion:

[0366] (1) Description: This invention innovatively integrates the visual features, text features, and layout features of UI design diagrams to construct a comprehensive multimodal information input. This enables the model to transcend the limitations of a single modality and understand the internal structure and function of the UI from a richer perspective.

[0367] (2) Beneficial effects: Compared with existing technologies that rely solely on visual features for segmentation, this invention can more accurately capture the semantic information of UI elements. For example, by combining the text content recognized by OCR (such as the word "Submit" on a button) with its visual shape, the "Submit" function can be accurately identified, rather than just a rectangular button. This significantly improves the accuracy and intelligence of UI design drawing segmentation, providing a more reliable semantic foundation for subsequent automated tasks.

[0368] 2. Context-based semantic region reasoning and aggregation:

[0369] (1) Description: Based on the identification of basic UI elements, this invention further performs contextual semantic analysis through a deep learning model, and intelligently infers and aggregates related elements according to the spatial relationships, logical relationships and text prompts between elements to form regions with clear semantics. For example, the search box, search button and search prompt text are aggregated into a "search area".

[0370] (2) Beneficial effects: Existing technologies often only perform isolated element identification or coarse-grained physical region division, making it difficult to form semantic regions corresponding to the actual functions of the UI. This invention introduces contextual reasoning and aggregation mechanisms, making the segmentation results more consistent with human understanding of UI functions. For example, it can identify functional modules such as "navigation bar", "product list", and "user comment area", greatly improving the usability and interpretability of the segmentation results and providing more direct and efficient input for intelligent code generation and UI automation testing.

[0371] 3. Supports flexible segmentation with multi-granularity and semantic matching:

[0372] (1) Description: This invention can flexibly output semantic segmentation results of different granularities according to different application needs (the user can specify the name of the element to be segmented). Whether it is a macro-level page area (such as header, body, footer), a meso-level functional module (such as search area, product card), or a micro-level interactive component (such as a single button, input box), it can provide accurate bounding boxes and semantic labels, and these segmentation results are highly matched with the actual semantic structure of the UI.

[0373] (2) Beneficial Effects: Existing technologies often have relatively fixed segmentation granularities, making it difficult to meet diverse application scenarios. This invention provides multi-granularity semantic matching segmentation capabilities, allowing developers and automation tools to select the most suitable granularity for operation based on specific tasks. For example, coarse-grained segmentation can be used when performing overall layout analysis. Results that do not meet the granularity requirements are filtered through rule validation (e.g., coarse-grained results must meet the requirements of "a single region contains ≥3 medium-grained modules" and "boundary box coverage ≥ 1 / 5 of the canvas"). If there is mixed-grained output (e.g., component-level tags mixed in with coarse-grained results), they are automatically removed or merged into the corresponding granularity tags. Fine-grained segmentation is used when performing specific component testing. This greatly enhances the flexibility and practicality of this method and reduces the complexity of secondary processing.

[0374] 4. Robustness to complex and diverse UIs:

[0375] (1) Description: This invention utilizes the powerful generalization ability of deep learning models and the comprehensiveness of multimodal features to maintain high-precision semantic segmentation capabilities when facing various design styles, complex layouts, and UI design diagrams containing custom components.

[0376] (2) Beneficial effects: Traditional methods are sensitive to changes in the style and structure of UI design diagrams and have poor generalization ability, requiring extensive customization or retraining for different UIs. This invention effectively overcomes this problem by learning the deep semantic rules of UI, significantly improving the robustness and versatility of the model, reducing deployment and maintenance costs, and enabling the method to be widely applied in various practical scenarios.

[0377] like Figure 9 As shown, this embodiment of the invention also provides a UI design diagram segmentation device 900 based on front-end semantic understanding, which can implement the above-described method. This device may include:

[0378] The first module 910 is used to obtain the UI design drawing to be segmented, preprocess the UI design drawing, and obtain the processed design drawing;

[0379] The second module 920 is used to extract features based on the processing design graph using a pre-trained multimodal deep neural network to obtain multimodal features; among which, multimodal features include visual features, text features and layout features;

[0380] The third module 930 is used to perform component semantic segmentation on the processing design graph based on multimodal features to obtain component data; wherein, the component data includes the original boundary and component category of each atomic-level component in the processing design graph;

[0381] The fourth module 940 is used to perform rule-based clustering of atomic-level components based on the original boundaries and component categories, thereby constructing a multi-level semantic hierarchy system;

[0382] The fifth module, 950, is used to output the UI design diagram segmentation results based on the multi-level semantic hierarchy.

[0383] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0384] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0385] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0386] like Figure 10 As shown, Figure 10 The hardware structure of an electronic device 1000 according to another embodiment is illustrated. The electronic device 1000 includes:

[0387] The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (aSIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0388] The memory 1002 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RaM). The memory 1002 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001.

[0389] Input / output interface 1003 is used to implement information input and output;

[0390] The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0391] Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004);

[0392] The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0393] The electronic device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0394] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0395] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0396] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0397] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0398] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0399] The UI design diagram segmentation method, apparatus, electronic device, storage medium, and program product based on front-end semantic understanding provided by this invention involves: acquiring a UI design diagram to be segmented; preprocessing the UI design diagram to obtain a processed design diagram; extracting features from the processed design diagram using a pre-trained multimodal deep neural network to obtain multimodal features, including visual features, text features, and layout features; performing component semantic segmentation on the processed design diagram based on the multimodal features to obtain component data, including the original boundary and component category of each atomic component in the processed design diagram; performing rule-based clustering on the atomic components based on the original boundary and component category to construct a multi-level semantic hierarchy; and outputting the UI design diagram segmentation result according to the multi-level semantic hierarchy. This invention significantly improves the semantic understanding of UI elements by introducing a multimodal deep neural network and integrating visual, textual, and layout features for joint analysis. It can accurately identify the functional meaning of atomic-level components. Furthermore, by constructing a multi-level semantic hierarchy through rule-based clustering, it effectively achieves semantic organization from atomic components to composite components, overcoming the problems of poor adaptability to complex UIs and difficulty in capturing implicit semantics in existing methods. Specifically, this invention can parse non-explicit semantic information based on context and text content, and achieve structural and semantic matching segmentation at different granularities, significantly improving the accuracy and usability of the segmentation results and providing a reliable foundation for advanced UI applications.

[0400] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0401] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0402] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0403] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0404] The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present invention should be within the scope of the claims of the present invention. < / nav>

Claims

1. A UI design diagram segmentation method based on front-end semantic understanding, characterized in that, The method includes the following steps: Obtain the UI design drawing to be segmented, preprocess the UI design drawing to obtain the processed design drawing; Based on the processing design diagram, feature extraction is performed using a pre-trained multimodal deep neural network to obtain multimodal features; wherein, the multimodal features include visual features, text features, and layout features; Based on the multimodal features, the processing design graph is segmented into components to obtain component data; wherein, the component data includes the original boundary and component category of each atomic-level component in the processing design graph; Based on the original boundaries and the component categories, the atomic-level components are clustered according to rules to construct a multi-level semantic hierarchy system; The UI design diagram segmentation result is output based on the multi-level semantic hierarchy system; Wherein, when the result of the feature extraction is the visual feature, the feature extraction using a pre-trained multimodal deep neural network includes the following steps: The convolutional layers of the pre-trained visual encoder are loaded as the original visual model, and the convolutional layers in the original visual model are frozen to initialize the initial visual model. The freezing operation includes a first number of convolutional layers before freezing and a second number of convolutional layers after freezing. The sum of the first number and the second number is the total number of convolutional layers in the initial visual model. The pre-trained visual encoder removes the original classification layer of the model and uses its convolutional layers to extract high-dimensional visual features of the image and load them as the original visual model. The initial visual model is trained using a design drawing labeled with visual features. An average classification accuracy is constructed based on the training feature map output by the penultimate convolutional layer in the initial visual model and the visual feature labels. The parameters of the initial visual model are adjusted based on the average classification accuracy to obtain the target visual model. The processing design diagram is input into the target visual model. The target feature map output by the penultimate convolutional layer in the target visual model is reduced in dimensionality and encoded by global average pooling and L2 normalization to obtain the visual features.

2. The method according to claim 1, characterized in that, The preprocessing of the UI design draft includes at least one of the following steps: Based on the target size, the UI design draft is scaled, and the mapping relationship between the original size of the UI design draft and the target size is recorded; Convert the UI design to a preset color space; The UI design drawing is subjected to image noise removal processing using Gaussian filtering; Sharpen the blurred areas of the UI design draft; The UI design draft is subjected to grayscale normalization.

3. The method according to claim 1, characterized in that, When the result of the feature extraction is the text feature, the feature extraction using a pre-trained multimodal deep neural network includes the following steps: The processing design diagram is subjected to text recognition based on a preset optical character recognition engine to obtain the recognized text; The bounding box of the text region corresponding to each identified text is located using a multi-scale text detection algorithm. The text content within the bounding box of the text region is identified using a recurrent neural network or a transformer model to obtain structured text; The structured text is input into a pre-trained large language model, and the text semantic embedding is obtained through forward propagation of the model as the text feature.

4. The method according to claim 1, characterized in that, When the result of the feature extraction is the layout feature, the feature extraction using a pre-trained multimodal deep neural network includes the following steps: Once the HTML source code of the UI design draft is obtained, layout features based on the DOM structure are extracted from the processed design draft. The layout feature extraction based on the DOM structure includes the following operations: A DOM tree structure is constructed based on the HTML source code; wherein, the DOM tree structure includes the element hierarchy of all DOM elements; The bounding box coordinates of each DOM element are obtained by converting the CSS style information in the HTML source code. The DOM elements are matched and identified by tag name and / or class name, and layout features are generated based on the key elements; wherein, the layout features include the bounding box coordinates of the key elements and the hierarchical relationship and number of sibling elements determined based on the element hierarchy; When the HTML source code of the UI design is not obtained, the layout features of the processed design are inferred based on visual analysis. The layout feature inference based on visual analysis includes the following operations: The processing design diagram is divided into text and non-text regions by combining optical character recognition with image segmentation algorithms. Each of the text regions is treated as a text element, and the bounding box coordinates of the text region are output based on the result of optical character recognition. Contour detection is performed on the non-text region, and then weak edges are strengthened by adaptive thresholding and edge enhancement to obtain the non-text elements and the corresponding bounding box coordinates. Based on the vertical and overlapping relationships of the text elements and non-text elements in the processing design diagram, the element hierarchy is obtained. The layout feature is generated based on the bounding box coordinates and the element hierarchy.

5. The method according to claim 1, characterized in that, The step of performing component semantic segmentation on the processing design graph based on the multimodal features to obtain component data includes the following steps: The atomic-level components and their corresponding original boundaries in the processing design diagram are obtained by using a semantic segmentation network. A lightweight classifier is used to determine the semantic label of the detection region corresponding to each original boundary, thereby obtaining the component category of each atomic component.

6. The method according to claim 1, characterized in that, The step of performing rule-based clustering on the atomic-level components to construct a multi-level semantic hierarchy includes the following steps: The atomic-level components are used as fine-grained regions at the component level, the original boundaries are used as component-level bounding boxes, and the component categories are used as component-level semantic labels. Based on the original boundary, the spatial relationship between each atomic-level component is determined, and then the component clustering based on the component category is performed to obtain a module-level medium-granularity region. Based on the region range marker module-level bounding boxes corresponding to the medium-granularity regions, semantic label determination is performed on the detection regions corresponding to the module-level bounding boxes to obtain module-level semantic labels for each medium-granularity region. The processing design diagram is divided into multiple large regions based on page layout logic using rule templates, and each medium-granular region is classified into the corresponding large region to obtain page-level coarse-granular regions. Based on the region range markers corresponding to the coarse-grained regions, page-level bounding boxes are marked, and semantic labels are determined for the detection regions corresponding to the page-level bounding boxes to obtain page-level semantic labels for each coarse-grained region.

7. A UI design drawing segmentation device based on front-end semantic understanding, characterized in that, The device includes: The first module is used to obtain the UI design drawing to be segmented, and to preprocess the UI design drawing to obtain the processed design drawing; The second module is used to extract features based on the processing design diagram using a pre-trained multimodal deep neural network to obtain multimodal features; wherein, the multimodal features include visual features, text features, and layout features; The third module is used to perform component semantic segmentation on the processing design graph based on the multimodal features to obtain component data; wherein, the component data includes the original boundary and component category of each atomic-level component in the processing design graph; The fourth module is used to perform rule-based clustering of the atomic-level components based on the original boundaries and the component categories to construct a multi-level semantic hierarchy system. The fifth module is used to output the UI design drawing segmentation result based on the multi-level semantic hierarchy system; Wherein, when the result of the feature extraction is the visual feature, the feature extraction using a pre-trained multimodal deep neural network includes the following steps: The convolutional layers of the pre-trained visual encoder are loaded as the original visual model, and the convolutional layers in the original visual model are frozen to initialize the initial visual model. The freezing operation includes a first number of convolutional layers before freezing and a second number of convolutional layers after freezing. The sum of the first number and the second number is the total number of convolutional layers in the initial visual model. The pre-trained visual encoder removes the original classification layer of the model and uses its convolutional layers to extract high-dimensional visual features of the image and load them as the original visual model. The initial visual model is trained using a design drawing labeled with visual features. An average classification accuracy is constructed based on the training feature map output by the penultimate convolutional layer in the initial visual model and the visual feature labels. The parameters of the initial visual model are adjusted based on the average classification accuracy to obtain the target visual model. The processing design diagram is input into the target visual model. The target feature map output by the penultimate convolutional layer in the target visual model is reduced in dimensionality and encoded by global average pooling and L2 normalization to obtain the visual features.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images