A method and apparatus for UI component analysis based on a large visual model

Abstract

This invention provides a method and apparatus for UI component analysis based on a large visual model, relating to the field of computer technology. The method includes: locating and segmenting a UI image using a large visual model to obtain coordinates and images of multiple component elements; classifying each element image using a deep residual network model; performing text recognition on each category of images using an optical character recognition model to obtain text features; processing each category of images to extract image features of each component element; extracting target coordinates corresponding to each component element from the coordinates of multiple component elements, and determining the nesting relationship between multiple component elements based on the target coordinates; and integrating the text features, image features, and nesting relationships to obtain integrated data. This allows for rapid segmentation and classification of UI images, improving the efficiency of UI component analysis, accurately identifying and analyzing relationships between components, and enhancing the completeness and accuracy of UI component analysis results.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for analyzing user interface (UI) components based on a large visual model. Background Technology

[0002] UI component analysis is a crucial step in front-end code development. A UI component encapsulates one or more code segments that perform their respective functions into one or more independent parts. These functional code segments together complete the UI representation. UI components are the fundamental elements for building a UI; a UI image contains multiple UI component elements. These UI component elements can be categorized in various ways to meet diverse design and interaction needs. UI component elements can be disassembled and reassembled in different ways to create a complete interface design.

[0003] Currently, UI component analysis involves using machine learning methods to train a UI structure parameter generation model based on a UI framework sample set; parsing the style image of the component to be generated to obtain UI image parameters for the user interface; inputting the UI image parameters into the UI structure parameter generation model to obtain UI framework structure parameters and interface style data; and matching the corresponding code in a pre-built codebase based on the UI framework structure parameters and interface style data to generate the component code corresponding to the UI to be generated. However, using machine learning for UI component analysis requires complex data annotation and model training processes, resulting in long analysis times and low efficiency. Since modern applications often have complex UI structures, such as nested component structures, the existing UI component analysis methods mentioned above neglect the understanding of element nesting relationships. For UI images with a large number of nested components, they cannot accurately identify and analyze the relationships between components, thus resulting in low accuracy in UI component analysis. Summary of the Invention

[0004] The purpose of this invention is to provide a method and apparatus for UI component analysis based on a large visual model, thereby solving the problems of low efficiency and low accuracy in UI component analysis.

[0005] To address the aforementioned technical problems, the embodiments of the present invention provide the following technical solutions:

[0006] The first aspect of this invention provides a UI component analysis method based on a large visual model, the method comprising:

[0007] Get the UI image, which contains multiple UI component elements;

[0008] A large visual model is used to locate and segment UI images, resulting in the coordinates of multiple component elements and multiple element images. Each element image contains at least one UI component element.

[0009] A deep residual network model is used to classify the images of each element, resulting in images of multiple categories;

[0010] An optical character recognition model is used to perform text recognition on images of various categories to obtain text features;

[0011] Image processing is performed on images of each category to extract the image features of each component element;

[0012] Extract the target coordinates of each component element from the coordinates of multiple component elements, and determine the nesting relationship between multiple component elements based on the target coordinates;

[0013] By integrating text features, image features, and nesting relationships, we obtain integrated data, which is then used for the automated development of front-end code for UI component elements.

[0014] In some modified embodiments of the first aspect of this application, after locating and segmenting the UI image using a large visual model to obtain the coordinates of multiple component elements and multiple element images, the method further includes:

[0015] Get the selected area and its corresponding coordinates. The selected area is the region containing the specified component element.

[0016] The overlap ratio (CR) algorithm is used to calculate the degree of overlap between the predicted region and the selected region corresponding to each element image.

[0017] The element images with an overlap of less than a preset value are filtered to obtain the filtered element images.

[0018] The selected area is segmented using point prompts to obtain the selected image corresponding to the selected area, and then the selected image and the filtered element images are integrated.

[0019] Based on the coordinates of the selected frame and the coordinates of the component elements corresponding to each filtered element image, the overlapping images in the integrated image and the size of the regions corresponding to each overlapping image are determined, and the images with smaller region sizes are removed to obtain the final segmented image.

[0020] In some modified embodiments of the first aspect of this application, the text features include multiple text coordinates, text size, text height, and text layout; the multiple component element coordinates include multiple text box coordinates; and an optical character recognition model is used to perform text recognition on images of various categories to obtain text features, including:

[0021] Calculate the text height based on the text box coordinates;

[0022] Calculate text size based on text coordinates;

[0023] The layout of the text is determined by the difference between the coordinates of the text box and the coordinates of the text.

[0024] In some modified embodiments of the first aspect of this application, the text layout includes left alignment, center alignment, and right alignment; the text box coordinates include the coordinates of the top-left corner and the top-right corner of the text box; the text coordinates include the coordinates of the top-left corner and the top-right corner of the text; and the text layout is determined based on the difference between the text box coordinates and the text coordinates, including:

[0025] Calculate the first difference between the top-left corner coordinates of the text box and the top-left corner coordinates of the text.

[0026] Calculate the second difference between the coordinates of the top-right corner of the text box and the coordinates of the top-right corner of the text.

[0027] When the first difference is less than the second difference, the text layout is determined to be left-aligned;

[0028] When the first difference equals the second difference, the text layout is set to center.

[0029] When the first difference is greater than the second difference, the text layout is determined to be right-aligned.

[0030] In some modified embodiments of the first aspect of this application, the image features include UI component element colors, text colors, UI component element border colors, and rectangle rounded corners.

[0031] In some modified embodiments of the first aspect of this application, image processing is performed on images of various categories to extract image features of each component element, including:

[0032] The background area of ​​each UI component element is determined based on its coordinates.

[0033] The background color of each UI component element is determined based on the RGB color of each pixel in the background area.

[0034] Convert each category of image into grayscale images, and use the big rule method to determine the threshold and convert the grayscale images into binary images;

[0035] Convert the color value corresponding to the background area color to the first grayscale value;

[0036] When the first grayscale value is greater than the threshold, the second grayscale value in the binary image that is less than the threshold is determined, and the RGB color corresponding to the second grayscale value is determined as the color of the UI component element.

[0037] When the first grayscale value is less than the threshold, a third grayscale value greater than the threshold is determined in the binary image, and the RGB color corresponding to the third grayscale value is determined as the color of the UI component element.

[0038] A second aspect of this application provides a UI component analysis device based on a large visual model, the device comprising:

[0039] The acquisition module is used to acquire UI images, which contain multiple UI component elements.

[0040] The UI component element segmentation module is used to locate and segment UI images using a large visual model, resulting in multiple component element coordinates and multiple element images, each of which contains at least one UI component element.

[0041] The UI component element classification module is used to classify the images of each element using a deep residual network model, resulting in images of multiple categories.

[0042] The text recognition module is used to perform text recognition on images of various categories using an optical character recognition model to obtain text features;

[0043] The image processing module is used to process images of various categories and extract the image features of each component element;

[0044] The relationship extraction module is used to extract the target coordinates corresponding to each component element from the coordinates of multiple component elements, and to determine the nesting relationship between multiple component elements based on the target coordinates;

[0045] The integration module is used to integrate text features, image features, and nesting relationships to obtain integrated data, which is then used for the automated development of front-end code for UI component elements.

[0046] In some modified embodiments of the second aspect of this application, the apparatus further includes a filtering module:

[0047] The filtering module is used to locate and segment the UI image using a large visual model, obtaining the coordinates of multiple component elements and multiple element images. It then obtains the selected region and its corresponding coordinates, where the selected region is the area containing the specified component element. Using the intersection-union algorithm, it calculates the overlap between the predicted region and the selected region for each element image. Element images with overlap less than a preset value are filtered to obtain filtered element images. Point-based cue segmentation is performed on the selected region to obtain the corresponding selected image, and the selected image and the filtered element images are integrated. Based on the selected coordinates and the component element coordinates corresponding to the filtered element images, the module determines the overlapping images in the integrated image and the size of the overlapping regions, removing images with smaller region sizes to obtain the final segmented image.

[0048] A third aspect of this application provides an electronic device, comprising: at least one processor; and at least one memory and a bus connected to the processor; wherein the processor and the memory communicate with each other via the bus; the processor is used to call program instructions in the memory to execute the UI component analysis method based on a visual large model described in the first aspect or any optional embodiment of the first aspect.

[0049] A fourth aspect of this application provides a computer-readable storage medium including a stored program, wherein, when the program is running, it controls the device where the storage medium is located to execute the UI component analysis method based on a visual large model described in the first aspect or any optional embodiment of the first aspect.

[0050] Compared to existing technologies, this invention provides a UI component analysis method and apparatus based on a large visual model. The method involves: acquiring a UI image containing multiple UI component elements; using a large visual model to locate and segment the UI image, obtaining coordinates and images of multiple component elements; classifying each element image using a deep residual network model, obtaining multiple category images; performing text recognition on each category image using an optical character recognition model, obtaining text features; performing image processing on each category image to extract image features of each component element; extracting target coordinates corresponding to each component element from the coordinates of multiple component elements, and determining the nesting relationship between multiple component elements based on the target coordinates; and integrating the text features, image features, and nesting relationships to obtain integrated data, which is used for automated front-end code development of UI component elements. In this way, a large visual model can be used to automatically locate and segment UI images, and a deep residual network model can be used to classify each element image. There is no need for overly complex data annotation, which can quickly segment and classify UI images, thereby shortening the time and improving the efficiency of UI component analysis. Extracting the target coordinates corresponding to each component element from multiple component element coordinates and determining the nesting relationship between multiple component elements based on the target coordinates can better handle complex UI images, accurately identify and analyze the relationship between components, and improve the completeness and accuracy of UI component analysis results. Attached Figure Description

[0051] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the invention are illustrated by way of example and not limitation, with the same or corresponding reference numerals denoteing the same or corresponding parts, wherein:

[0052] Figure 1 The flowchart illustrates the process of UI component analysis based on a visual big model. Figure 1 ;

[0053] Figure 2 The flowchart illustrates the process of UI component analysis based on a visual big model. Figure 2 ;

[0054] Figure 3 A schematic diagram of the UI component analysis device based on a large visual model is shown.

[0055] Figure 4 A schematic diagram of the electronic device is shown. Detailed Implementation

[0056] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the invention and to fully convey the scope of the invention to those skilled in the art.

[0057] It should be noted that, unless otherwise stated, the technical or scientific terms used in this invention should have the ordinary meaning as understood by those skilled in the art.

[0058] Current technologies using machine learning for UI component analysis require complex data annotation and model training processes, resulting in lengthy analysis times and low efficiency. Furthermore, they neglect understanding element nesting relationships, making it difficult to accurately identify and analyze these relationships in UI images with numerous nested components, thus further reducing the accuracy of the analysis. Therefore, a more efficient and accurate method for UI component analysis is needed.

[0059] This invention acquires a UI image containing multiple UI component elements; it uses a large visual model to locate and segment the UI image, obtaining coordinates and images of multiple component elements; it uses a deep residual network model to classify the element images, obtaining multiple category images; it uses an optical character recognition model to perform text recognition on each category of images, obtaining text features; it performs image processing on each category of images to extract image features of each component element; it extracts the target coordinates corresponding to each component element from the coordinates of multiple component elements, and determines the nesting relationship between multiple component elements based on the target coordinates; it integrates the text features, image features, and nesting relationships to obtain integrated data, which is used for the automated development of front-end code for UI component elements.

[0060] In this way, a large visual model can be used to automatically locate and segment UI images, and a deep residual network model can be used to classify each element image. There is no need for overly complex data annotation, which can quickly segment and classify UI images, thereby shortening the time and improving the efficiency of UI component analysis. Extracting the target coordinates corresponding to each component element from multiple component element coordinates and determining the nesting relationship between multiple component elements based on the target coordinates can better handle complex UI images, accurately identify and analyze the relationship between components, and improve the completeness and accuracy of UI component analysis results.

[0061] The main idea of ​​this invention is to use a large visual model to automatically segment images, extract all elements from the image, and then perform automatic recognition and feature extraction to understand the nesting relationships between UI elements, thereby enabling the analysis of all UI component elements in the entire UI image.

[0062] The methods in the embodiments of the present invention will be described in detail below.

[0063] Figure 1 This schematically illustrates an embodiment of the invention targeting...

[0064] See the flowchart of the method. Figure 1 As shown, the method may include:

[0065] S101. Obtain UI image.

[0066] The UI image contains multiple UI component elements.

[0067] Multiple UI component elements can be the text in the input box, the outer frame of the input box, etc.

[0068] S102. Use a large visual model to locate and segment the UI image to obtain the coordinates of multiple component elements and multiple element images.

[0069] Each element image contains at least one UI component element.

[0070] Specifically, the Segment Anything Model (SAM) is used to locate and segment the UI image obtained in step S101 to obtain the coordinates of multiple component elements and multiple element images. Each element image may contain one UI component element or multiple UI component elements.

[0071] The coordinates of multiple component elements include the coordinates of the text box.

[0072] S103. A deep residual network model is used to classify the images of each element, resulting in images of multiple categories.

[0073] Specifically, a Deep Residual Network (ResNet) model is used to classify the element images obtained in step S102, resulting in multiple category images.

[0074] A category image contains UI component elements of the same category. The UI component element categories can be input box category, button category, counter category, etc.

[0075] S104. Use an optical character recognition model to perform text recognition on images of various categories to obtain text features.

[0076] Specifically, the Optical Character Recognition (OCR) model is used to perform text recognition on the images of each category obtained in step S103 to obtain text features.

[0077] Text features include multiple text coordinates, text size, text height, and text layout.

[0078] S105. Perform image processing on images of each category to extract the image features of each component element.

[0079] Specifically, image processing is performed on the images of each category obtained in step S103 to extract the image features of each component element.

[0080] Image processing includes UI component element color recognition, text color recognition, UI component element border color recognition, and rectangle rounded corner determination.

[0081] Image features include UI component element colors, text colors, UI component element border colors, and rectangle rounded corners.

[0082] S106. Extract the target coordinates corresponding to each component element from the coordinates of multiple component elements, and determine the nesting relationship between multiple component elements based on the target coordinates.

[0083] Specifically, the target coordinates corresponding to each component element are extracted from the multiple component element coordinates obtained in step S102, and the nesting relationship between the multiple component elements is determined based on the target coordinates.

[0084] The target coordinates can be the coordinates of the top-left corner (top-left point), bottom-left corner (bottom-left point), top-right corner (top-right point), and bottom-right corner (bottom-right point) among all the coordinates of the components enclosed. Based on the top-left, bottom-left, top-right, and bottom-right coordinates of multiple component elements, the nesting relationship between them is determined.

[0085] S107. Integrate text features, image features, and nesting relationships to obtain integrated data.

[0086] Among them, the integrated data is used for the automated development of front-end code for UI component elements.

[0087] Specifically, the text features obtained in step S104, the image features extracted in step S105, and the nesting relationships determined in step S106 are integrated. In other words, the text features obtained in step S104, the image features extracted in step S105, and the nesting relationships determined in step S106 are put into a file to obtain integrated data for the automated development of front-end code for UI component elements.

[0088] Based on the above Figure 1 As can be seen from the implementation method, the embodiments of the present invention acquire a UI image, which contains multiple UI component elements; a large visual model is used to locate and segment the UI image to obtain the coordinates of multiple component elements and multiple element images; a deep residual network model is used to classify each element image to obtain multiple category images; an optical character recognition model is used to perform text recognition on each category image to obtain text features; image processing is performed on each category image to extract the image features of each component element; target coordinates corresponding to each component element are extracted from the coordinates of multiple component elements, and the nesting relationship between multiple component elements is determined based on the target coordinates; the text features, image features, and nesting relationships are integrated to obtain integrated data, which is used for the automated development of the front-end code of UI component elements. In this way, a large visual model can be used to automatically locate and segment UI images, and a deep residual network model can be used to classify each element image. There is no need for overly complex data annotation, which can quickly segment and classify UI images, thereby shortening the time and improving the efficiency of UI component analysis. Extracting the target coordinates corresponding to each component element from multiple component element coordinates and determining the nesting relationship between multiple component elements based on the target coordinates can better handle complex UI images, accurately identify and analyze the relationship between components, and improve the completeness and accuracy of UI component analysis results.

[0089] As a refinement and extension of the above embodiments, Figure 2 This is a flowchart of a UI component analysis method based on a large visual model, as described in an embodiment of the present invention. Figure 2 See Figure 2 As shown in the figure, the UI component analysis method based on a large visual model provided by this invention may include:

[0090] S201. Obtain UI image.

[0091] The UI image contains multiple UI component elements.

[0092] Step S201 is the same as step S101, so it will not be described again here.

[0093] S202. Use a large visual model to locate and segment the UI image to obtain the coordinates of multiple component elements and multiple element images.

[0094] Each element image contains at least one UI component element.

[0095] Step S202 is the same as step S102, so it will not be described again here.

[0096] Using SAM to segment images can distinguish the component elements in UI images without pre-setting the cutting size, making segmentation more flexible and convenient. SAM can be used to automatically segment UI images, separating all component elements, followed by automatic recognition and feature extraction, directly analyzing the entire UI image.

[0097] S203. Obtain the selected area and the corresponding coordinates of the selected area.

[0098] The selected area is the region containing the specified component element.

[0099] The selected area is the region containing the specified component element. It can be a region that the user wants to divide in the UI drawing, along with its corresponding coordinates. Providing the user with a region to be divided in the UI drawing, along with its coordinates, can also be understood as the user providing a hint.

[0100] By using SAM to automatically locate and segment the UI image, and obtaining the coordinates of multiple component elements and multiple element images, we can obtain the selection area that the user wants to segment in the UI image, as well as the corresponding selection area coordinates. Based on the user prompts, we can segment all the components in the UI image to obtain the selection area, ensuring the integrity of component segmentation.

[0101] S204. Using the cross-union algorithm, calculate the overlap between the predicted region and the selected region corresponding to each element image.

[0102] Specifically, the Intersection over Union (IoU) algorithm is used to calculate the overlap between the predicted regions corresponding to each element image obtained in step S202 and the selected regions obtained in step S203.

[0103] The prediction region corresponding to each element image is the region occupied by each element image.

[0104] S205. Filter the element images corresponding to those with an overlap of less than a preset value to obtain the filtered element images.

[0105] Specifically, based on the overlap between the predicted region and the selected region corresponding to each element image, multiple element images are filtered, and the element images corresponding to the overlap calculated in step S204 being less than the preset value are filtered to obtain the filtered element images.

[0106] The preset values ​​can be 30%, 50%, or 70%.

[0107] Images of elements with an overlap ratio equal to or greater than a preset value will not be filtered.

[0108] S206. Perform point-based segmentation on the selected area to obtain the selected image corresponding to the selected area, and integrate the selected image with the filtered images of each element.

[0109] Specifically, the selected area obtained in step S203 is segmented using point prompts to obtain the selected image corresponding to the selected area, and the selected image is integrated with the filtered element images obtained in step S205.

[0110] S207. Based on the coordinates of the selected frame and the coordinates of the component elements corresponding to each element image after filtering, determine the overlapping images in the integrated image and the size of the region corresponding to each overlapping image, and remove the images with smaller region sizes to obtain the final segmented image.

[0111] In the integrated image, overlapping images have a higher degree of overlap than those in the filtered element images. By using the bounding box coordinates and the corresponding component element coordinates of the filtered element images, the regions of each image in the integrated image are determined. It is then determined whether these regions overlap. If they do, the size of the overlapping regions is determined, and the image with the larger region size is retained, while the image with the smaller region size is removed, resulting in the final segmented image. This solves the problem that bounding box segmentation can only segment one UI component element at a time, and also addresses the issue of automatic segmentation generating a large number of fragmented images.

[0112] For irregular UI component elements that are difficult to segment using bounding boxes, a lasso technique—using a series of points—can be used for segmentation. By processing the series of points surrounding the UI component element and calculating their coordinates, the maximum and minimum XY coordinates are obtained. This makes the series of points surrounding the UI component element the smallest bounding box, which becomes the bounding box tooltip in the SAM model. Points within the UI component element are then used as point tools in the SAM model. The processed bounding box tooltip, point tooltip, and UI image are then used as parameters in conjunction with the SAM model to segment the UI component element, outputting a contour map of the irregular UI component element, thus solving the segmentation problem for irregular UI component elements.

[0113] S208. A deep residual network model is used to classify the images of each element, resulting in images of multiple categories.

[0114] Specifically, the ResNet model is used to classify the final segmented image obtained in step S207, resulting in images of multiple categories.

[0115] A category image contains UI component elements of the same category. The UI component element categories can be input box category, button category, counter category, etc.

[0116] Using SAM and ResNet models to segment and classify UI component elements respectively, this replaces traditional object detection algorithms and achieves better results. Only a small number of UI images are needed to train the model in the early stage to achieve good classification results, making the model training cost lower.

[0117] S209. Use an optical character recognition model to perform text recognition on images of various categories to obtain text features.

[0118] Text features include multiple text coordinates, text size, text height, and text layout.

[0119] Specifically, the coordinates of multiple component elements include the coordinates of multiple text boxes. An optical character recognition model is used to perform text recognition on images of various categories to obtain text features, including:

[0120] Step A1: Calculate the text height based on the text box coordinates.

[0121] The text height is calculated by using the coordinates of the text box between the two lines.

[0122] Step A2: Calculate the text size based on the text coordinates.

[0123] The text height is calculated by using the text coordinates between the two lines.

[0124] Step A3: Determine the text layout based on the difference between the text box coordinates and the text coordinates.

[0125] The text layout options include left alignment, center alignment, and right alignment. The text box coordinates include the coordinates of the top-left corner and the top-right corner of the text box. The text coordinates include the coordinates of the top-left corner and the top-right corner of the text.

[0126] The text layout is determined based on the difference between the text box coordinates and the text coordinates, including:

[0127] Step A31: Calculate the first difference between the coordinates of the top left corner of the text box and the coordinates of the top left corner of the text.

[0128] Step A32: Calculate the second difference between the coordinates of the top right corner of the text box and the coordinates of the top right corner of the text.

[0129] Step A33: When the first difference is less than the second difference, determine that the text layout is left-aligned.

[0130] Step A34: When the first difference equals the second difference, determine the text layout as centered.

[0131] Step A35: When the first difference is greater than the second difference, determine that the text layout is right-aligned.

[0132] S210. Perform image processing on images of each category to extract the image features of each component element.

[0133] Image processing includes UI component element color recognition, text color recognition, UI component element border color recognition, and rectangle rounded corner determination.

[0134] Image features include UI component element colors, text colors, UI component element border colors, and rectangle rounded corners.

[0135] Image processing is performed on images of each category to extract image features of each component element, including: element color recognition for images of each category to extract the UI component element color of each component element; text color recognition for images of each category to extract the text color of each component element; UI component element border color recognition for images of each category to extract the UI component element border color of each component element; and rectangle rounded corner determination for images of each category to extract the rectangle rounded corner of each component element.

[0136] For example, each category of image can be an image containing elements such as buttons, input boxes, counters, etc. Border color recognition and rounded corner recognition are performed on the images of elements such as buttons, input boxes, counters, etc. Component color recognition is performed on elements such as radio buttons and checkboxes, and text color recognition is performed on text blocks, etc., to obtain the feature information of each type of element.

[0137] Specifically, element color recognition is performed on images of each category to extract the colors of UI component elements, including:

[0138] Step B1: Determine the background area of ​​each UI component element based on its coordinates.

[0139] Step B2: Determine the background area color of each UI component element based on the RGB color of each pixel in the background area.

[0140] Step B3: Convert each category of image into grayscale image, and use the big rule to determine the threshold and convert the grayscale image into a binary image.

[0141] The method utilizes a large-scale law to determine the threshold and convert the grayscale image into a binary image. Specifically, firstly, the number of pixels at each grayscale level in the grayscale image is counted to form a grayscale histogram; then, the grayscale histogram is normalized to calculate the relative probability of each grayscale level; the inter-class variance when the grayscale image is divided into two categories is calculated; finally, the threshold for the maximum inter-class variance is found, and the image is divided into two categories: foreground and background of UI component elements, thus converting the grayscale image into a binary image that includes both foreground and background UI component elements.

[0142] Step B4: Convert the color value corresponding to the background area color to the first grayscale value.

[0143] Step B5: When the first grayscale value is greater than the threshold, determine the second grayscale value in the binary image that is less than the threshold, and determine the RGB color corresponding to the second grayscale value as the color of the UI component element.

[0144] The first grayscale value is compared with the threshold determined by the law. If the first grayscale value is greater than the threshold, the second grayscale value in the binary image that is less than the threshold is determined, and the RGB color corresponding to the second grayscale value is determined as the color of the UI component element.

[0145] When the first grayscale value is greater than the threshold, the second grayscale value in the binary image that is less than the threshold is determined, and the second grayscale value in the binary image is converted into RGB color using the big law method. This RGB color is then determined as the color of the UI component element.

[0146] Step B6: When the first grayscale value is less than the threshold, determine the third grayscale value in the binary image that is greater than the threshold, and determine the RGB color corresponding to the third grayscale value as the color of the UI component element.

[0147] The first grayscale value is compared with the threshold determined by the law. If the first grayscale value is less than the threshold, a third grayscale value greater than the threshold is determined in the binary image, and the RGB color corresponding to the third grayscale value is determined as the color of the UI component element.

[0148] When the first grayscale value is less than the threshold, a third grayscale value greater than the threshold is determined in the binary image, and the third grayscale value in the binary image is converted into an RGB color using the big law method. This RGB color is then determined as the color of the UI component element.

[0149] Specifically, text color recognition is performed on images of each category to extract the text color of each component element, including:

[0150] Step C1: Use the area outside the text box coordinates but within the component element coordinates given by the OCR model as the text background area.

[0151] Step C2: Determine the background area color of each UI component element based on the RGB color of each pixel in the text background area.

[0152] Step C3: Convert each category of image into grayscale image, and use the big rule to determine the threshold and convert the grayscale image into a binary image.

[0153] The method utilizes a large-scale law to determine the threshold and convert the grayscale image into a binary image. Specifically, firstly, the number of pixels at each grayscale level in the grayscale image is counted to form a grayscale histogram; then, the grayscale histogram is normalized to calculate the relative probability of each grayscale level; the inter-class variance when the grayscale image is divided into two categories is calculated; finally, the threshold for the maximum inter-class variance is found, and the image is divided into two categories: foreground and background of UI component elements, thus converting the grayscale image into a binary image that includes both foreground and background UI component elements.

[0154] Step C4: Convert the color value corresponding to the background area color to grayscale value.

[0155] Step C5: When the grayscale value is greater than the threshold, determine the grayscale values ​​in the binary image that are less than the threshold, and determine the RGB colors corresponding to the grayscale values ​​in the binary image that are less than the threshold as the colors of the UI component elements.

[0156] The grayscale value is compared with the threshold determined by the law. When the grayscale value is greater than the threshold, the grayscale value in the binary image that is less than the threshold is determined, and the RGB color corresponding to the grayscale value in the binary image that is less than the threshold is determined as the color of the UI component element.

[0157] When the grayscale value is greater than the threshold, determine the grayscale values ​​in the binary image that are less than the threshold, and use the big law to reverse the grayscale values ​​in the binary image that are less than the threshold to convert them into RGB colors, and then determine the RGB colors as the colors of UI component elements.

[0158] Step C6: When the grayscale value is less than the threshold, determine the grayscale values ​​in the binary image that are greater than the threshold, and determine the RGB colors corresponding to the grayscale values ​​in the binary image that are greater than the threshold as the colors of the UI component elements.

[0159] The grayscale value is compared with the threshold determined by the law. When the grayscale value is less than the threshold, the grayscale value in the binary image that is greater than the threshold is determined, and the RGB color corresponding to the grayscale value that is greater than the threshold in the binary image is determined as the color of the UI component element.

[0160] When the grayscale value is less than the threshold, determine the grayscale values ​​in the binary image that are greater than the threshold, and use the big law to reverse the grayscale values ​​in the binary image that are greater than the threshold to convert them into RGB colors, and then determine the RGB colors as the colors of UI component elements.

[0161] Specifically, the UI component element border colors are identified for each category of images, and the UI component element border colors for each component element are extracted, including:

[0162] Using the Canny edge detection algorithm from image processing, image noise is first removed by Gaussian filtering. Then, gradient calculation is performed to obtain a gradient image. Next, non-maximum points in the gradient image are suppressed to eliminate non-edge pixels. Finally, two thresholds are selected (pixels below the lower threshold are identified as false edges, and pixels above the higher threshold are identified as strong edges) to detect and connect image edges. After obtaining the image edges, the findContours algorithm from OpenCV is used to find the first boundary point (i.e., the edge obtained by the previous edge detection algorithm) in the image with the obtained edges, from top to bottom and from left to right. Then, it continuously searches for another pixel with the same value as the current pixel until all points form a contour, i.e., the outermost contour of the UI component element. Afterwards, the coordinates of the UI component element's border are located, and the average value of the pixels corresponding to the UI component element's border color is calculated to identify the UI component element's border color.

[0163] Edge detection algorithms can be the Canny algorithm, or other edge detection algorithms besides Canny. Contour detection algorithms can be the findContours algorithm, or other contour detection algorithms besides findContours.

[0164] Specifically, the rounded corners of rectangles are determined for each category of images, and the rounded corners of the rectangles of each component element are extracted, including:

[0165] Using the Canny edge detection algorithm from image processing, image noise is first removed by Gaussian filtering. Then, gradient calculation is performed to obtain a gradient image. After obtaining the gradient image, non-maximum points in the gradient image are suppressed to eliminate non-edge pixels. Finally, two thresholds are selected (pixels below the lower threshold are identified as false edges, and pixels above the higher threshold are identified as strong edges) to detect and connect image edges. After obtaining the image edges, the findContours algorithm from OpenCV is used to find the first boundary point in the image from top to bottom and left to right (i.e., the edge obtained by the previous edge detection algorithm). Then, it continuously searches for another pixel with the same value as the current pixel until all points are connected to form a contour, which is the outermost contour of the UI component element. Then, polygon fitting is performed on the found contour, and the angle of its intersection is calculated. If the angle is between 85° and 95°, it is a right angle; otherwise, it is determined whether the rectangle is a rounded rectangle.

[0166] Edge detection algorithms can be the Canny algorithm, or other edge detection algorithms besides Canny. Contour detection algorithms can be the findContours algorithm, or other contour detection algorithms besides findContours.

[0167] The OCR model was used to perform text recognition on some elements containing text to obtain the text content in the elements. At the same time, a series of image processing methods were used for different types of UI component elements to obtain features such as UI component element color, text color, UI component element border color, and rectangle rounded corners, so as to extract different feature information for different types of UI component elements. Personalized feature recognition was performed on different categories of UI component elements, resulting in rich UI component element features.

[0168] S211. Extract the target coordinates corresponding to each component element from the coordinates of multiple component elements, and determine the nesting relationship between multiple component elements based on the target coordinates.

[0169] Specifically, the target coordinates corresponding to each component element are extracted from the multiple component element coordinates obtained in step S202, and the nesting relationship between the multiple component elements is determined based on the target coordinates.

[0170] The target coordinates can be the coordinates of the top left corner (top left point), the bottom left corner (bottom left point), the top right corner (top right point), and the bottom right corner (bottom right point) among all the coordinates of the components.

[0171] The connection boxes formed by connecting the top-left, bottom-left, top-right, and bottom-right coordinates of each pair of component elements are compared. If one connection box is completely contained within another connection box, that is, if one UI component element is completely contained within another UI component element, then the two UI component elements are determined to be nested. If one connection box is not completely contained within another connection box, then the two UI component elements are determined not to be nested.

[0172] The nesting relationships of UI components are crucial for UI design. They enable better handling of complex UI images, accurate identification and analysis of relationships between components, and improved completeness and accuracy of UI component analysis results.

[0173] S212. Integrate text features, image features, and nesting relationships to obtain integrated data.

[0174] Among them, the integrated data is used for the automated development of front-end code for UI component elements.

[0175] Based on the same inventive concept, as an implementation of the above-mentioned UI component analysis method based on a large visual model, this embodiment of the invention also provides a UI component analysis device based on a large visual model.

[0176] Figure 3 This is a structural diagram of the device in an embodiment of the present invention. See also: Figure 3 As shown, the device may include:

[0177] Module 301 is used to acquire a UI image, which contains multiple UI component elements;

[0178] UI component element segmentation module 302 is used to locate and segment UI images using a large visual model to obtain multiple component element coordinates and multiple element images, each element image containing at least one UI component element.

[0179] UI component element classification module 303 is used to classify each element image using a deep residual network model to obtain multiple category images;

[0180] The text recognition module 304 is used to perform text recognition on images of various categories using an optical character recognition model to obtain text features;

[0181] Image processing module 305 is used to perform image processing on images of various categories and extract image features of each component element;

[0182] The relationship extraction module 306 is used to extract the target coordinates corresponding to each component element from the coordinates of multiple component elements, and to determine the nesting relationship between multiple component elements based on the target coordinates;

[0183] Integration module 307 is used to integrate text features, image features, and nesting relationships to obtain integrated data, which is used for the automated development of front-end code for UI component elements.

[0184] The acquisition module 301, the UI component element segmentation module 302, and the UI component element classification module 303 can be collectively referred to as the UI component element recognition sub-device. The text recognition module 304 and the image processing module 305 can be collectively referred to as the UI component element feature extraction sub-device, and the relationship extraction module 306 and the integration module 307 can be collectively referred to as the UI component element relationship understanding sub-device.

[0185] The device may also include a filter module.

[0186] The filtering module is used to locate and segment the UI image using a large visual model, obtaining the coordinates of multiple component elements and multiple element images. It then obtains the selected region and its corresponding coordinates, where the selected region is the area containing the specified component element. Using the intersection-union algorithm, it calculates the overlap between the predicted region and the selected region for each element image. Element images with overlap less than a preset value are filtered to obtain filtered element images. Point-based cue segmentation is performed on the selected region to obtain the corresponding selected image, and the selected image and the filtered element images are integrated. Based on the selected coordinates and the component element coordinates corresponding to the filtered element images, the module determines the overlapping images in the integrated image and the size of the overlapping regions, removing images with smaller region sizes to obtain the final segmented image.

[0187] The text recognition module 304 is specifically used to calculate the text height based on the text box coordinates; calculate the text size based on the text coordinates; and determine the text layout based on the difference between the text box coordinates and the text coordinates. The text features include multiple text coordinates, text size, text height, and text layout. The coordinates of the multiple component elements include multiple text box coordinates.

[0188] The text recognition module 304 is specifically used to calculate the first difference between the coordinates of the top left corner of the text box and the coordinates of the top left corner of the text; calculate the second difference between the coordinates of the top right corner of the text box and the coordinates of the top right corner of the text; when the first difference is less than the second difference, determine that the text layout is left-aligned; when the first difference is equal to the second difference, determine that the text layout is centered; when the first difference is greater than the second difference, determine that the text layout is right-aligned; the text layout includes left alignment, center alignment, and right alignment, and the text box coordinates include the coordinates of the top left corner of the text box and the coordinates of the top right corner of the text box.

[0189] The image processing module 305 is specifically used for image features including UI component element color, text color, UI component element border color, and rectangle rounded corners.

[0190] The image processing module 305 is specifically used to: determine the background area of ​​each UI component element based on the coordinates of each component element; determine the background area color of each UI component element based on the RGB color corresponding to each pixel in the background area; convert each type of image into a grayscale image, and use a large-scale law to determine the threshold and convert the grayscale image into a binary image; convert the color value corresponding to the background area color into a first grayscale value; when the first grayscale value is greater than the threshold, determine a second grayscale value in the binary image that is less than the threshold, and determine the RGB color corresponding to the second grayscale value as the UI component element color; when the first grayscale value is less than the threshold, determine a third grayscale value in the binary image that is greater than the threshold, and determine the RGB color corresponding to the third grayscale value as the UI component element color.

[0191] Based on the same inventive concept, embodiments of the present invention also provide an electronic device. Figure 4 This is a structural diagram of the electronic device in an embodiment of the present invention. See also... Figure 4 As shown, the electronic device 40 may include: at least one processor 401; and at least one memory 402 and bus 403 connected to the processor 401; wherein the processor 401 and the memory 402 communicate with each other through the bus 403; the processor 401 is used to call program instructions in the memory 402 to execute the UI component analysis method based on the visual large model in one or more of the above embodiments.

[0192] It should be noted that the above description of the user interface (UI) component analysis device based on a large visual model is similar to the description of the method embodiment described above, and has similar beneficial effects. For technical details not disclosed in the embodiments of the user interface (UI) component analysis device based on a large visual model of the present invention, please refer to the description of the method embodiment of the present invention for understanding.

[0193] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium, the computer-readable storage medium including a stored program, wherein, when the program is running, it controls the device where the storage medium is located to execute the methods in one or more of the above embodiments.

[0194] It should be noted that the descriptions of the above computer-readable storage medium embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the embodiments of the computer-readable storage medium of this invention, please refer to the descriptions of the method embodiments of this invention for understanding.

[0195] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A UI component analysis method based on a large visual model, characterized in that, The method includes: Obtain a UI image, which contains multiple UI component elements; The UI image is located and segmented using a large visual model to obtain the coordinates of multiple component elements and multiple element images, each element image containing at least one UI component element. Get the selected area and its corresponding coordinates, where the selected area is the region containing the specified component element; The overlap ratio between the predicted region and the selected region corresponding to each element image is calculated using the intersection-union algorithm. The element images with an overlap of less than a preset value are filtered to obtain the filtered element images. The selected area is segmented using point prompts to obtain the selected image corresponding to the selected area, and the selected image and the filtered element images are integrated. Based on the selected coordinates and the component element coordinates corresponding to each filtered element image, determine the overlapping images in the integrated image and the size of the region corresponding to each overlapping image, and remove the images with smaller region sizes to obtain the final segmented image. A deep residual network model is used to classify the images of each element, resulting in images of multiple categories; An optical character recognition model is used to perform text recognition on images of various categories to obtain text features; Image processing is performed on the images of each category to extract the image features of each component element; Extract the target coordinates corresponding to each component element from the coordinates of the plurality of component elements, and determine the nesting relationship between the plurality of component elements based on the target coordinates; The text features, image features, and nesting relationships are integrated to obtain integrated data, which is used for the automated development of front-end code for UI component elements. The image processing of the images of each category, extracting the image features of each component element, includes: Element color recognition is performed on the images of each category to extract the UI component element color of each component element; text color recognition is performed on the images of each category to extract the text color of each component element; UI component element border color recognition is performed on the images of each category to extract the UI component element border color of each component element; and rectangle rounded corner judgment is performed on the images of each category to extract the rectangle rounded corner of each component element. The method further includes: For multiple irregular UI component elements, a series of points surrounding the irregular UI component elements are processed. By calculating the coordinates of the series of points, the largest and smallest XY coordinates among the points are obtained, so that the series of points are converted into the smallest bounding box that encloses the multiple irregular UI component elements. The smallest bounding box is then converted into a box tooltip of the SAM model. The points within the multiple irregular UI component elements are used as point tools of the SAM model. The box tools, point tools, and UI image are used as parameters and input into the SAM model to segment the multiple irregular UI component elements and output the outline map of the multiple irregular UI component elements.

2. The method according to claim 1, characterized in that, The text features include multiple text coordinates, text size, text height, and text layout. The multiple component element coordinates include multiple text box coordinates. The text features are obtained by using an optical character recognition model to perform text recognition on images of various categories, including: Calculate the text height based on the coordinates of the text box; Calculate the text size based on the text coordinates; The layout of the text is determined by the difference between the coordinates of the text box and the coordinates of the text.

3. The method according to claim 2, characterized in that, The text layout includes left alignment, center alignment, and right alignment. The text box coordinates include the coordinates of the top-left corner and the top-right corner of the text box. The text coordinates include the coordinates of the top-left corner and the top-right corner of the text. Determining the text layout based on the difference between the text box coordinates and the text coordinates includes: Calculate the first difference between the coordinates of the top left corner of the text box and the coordinates of the top left corner of the text. Calculate the second difference between the coordinates of the top right corner of the text box and the coordinates of the top right corner of the text. When the first difference is less than the second difference, the layout of the text is determined to be left-aligned; When the first difference equals the second difference, the layout of the text is determined to be centered. When the first difference is greater than the second difference, the layout of the text is determined to be right-aligned.

4. The method according to claim 1, characterized in that, The image features include UI component element colors, text colors, UI component element border colors, and rectangle rounded corners.

5. The method according to claim 4, characterized in that, The image processing of the images of each category, extracting the image features of each component element, includes: Based on the coordinates of each component element, the background area of ​​each UI component element is determined; The background color of each UI component element is determined based on the RGB color of each pixel in the background area. The images of each category are converted into grayscale images, and the threshold is determined using the big rule method, and the grayscale images are converted into binary images. Convert the color value corresponding to the background area color into a first grayscale value; When the first grayscale value is greater than the threshold, a second grayscale value less than the threshold is determined in the binary image, and the RGB color corresponding to the second grayscale value is determined as the color of the UI component element; When the first grayscale value is less than the threshold, a third grayscale value greater than the threshold is determined in the binary image, and the RGB color corresponding to the third grayscale value is determined as the color of the UI component element.

6. A UI component analysis device based on a large visual model, characterized in that, The device includes: The acquisition module is used to acquire a UI image, which contains multiple UI component elements; The UI component element segmentation module is used to locate and segment the UI image using a large visual model to obtain multiple component element coordinates and multiple element images, each element image containing at least one UI component element. The UI component element classification module is used to classify the images of each element using a deep residual network model to obtain multiple category images; The text recognition module is used to perform text recognition on images of various categories using an optical character recognition model to obtain text features; The image processing module is used to perform image processing on the images of each category and extract the image features of each component element; The relationship extraction module is used to extract the target coordinates corresponding to each component element from the coordinates of the multiple component elements, and to determine the nesting relationship between the multiple component elements based on the target coordinates; An integration module is used to integrate the text features, the image features, and the nesting relationship to obtain integrated data, which is used for the automated development of front-end code for UI component elements; The device also includes a filtering module: The filtering module is used to locate and segment the UI image using a large visual model to obtain multiple component element coordinates and multiple element images. It then obtains a bounding box region and its corresponding bounding box coordinates, where the bounding box region is a region containing a specified component element. Using an intersection-union algorithm, it calculates the overlap between the predicted region and the bounding box region corresponding to each element image. Element images with overlap less than a preset value are filtered to obtain filtered element images. Point-based cue segmentation is performed on the bounding box region to obtain the bounding box image corresponding to the bounding box region. The bounding box image and the filtered element images are then integrated. Based on the bounding box coordinates and the component element coordinates corresponding to the filtered element images, the overlapping images in the integrated image and the size of the corresponding regions of the overlapping images are determined. Images with smaller region sizes are removed to obtain the final segmented image. The image processing module is specifically used to perform element color recognition on the images of each category, extract the UI component element color of each component element, perform text color recognition on the images of each category, extract the text color of each component element, perform UI component element border color recognition on the images of each category, extract the UI component element border color of each component element, and perform rectangle rounded corner determination on the images of each category, extract the rectangle rounded corner of each component element. The device further includes: processing a series of points surrounding multiple irregular UI component elements, calculating the coordinates of the series of points to obtain the largest and smallest XY coordinates among the series of points, so that the series of points are converted into a minimum bounding box that encloses the multiple irregular UI component elements, so that the minimum bounding box is converted into a box tooltip of the SAM model, and using the points within the multiple irregular UI component elements as point tools of the SAM model, and inputting the box tooltip, the point tooltip, and the UI image as parameters into the SAM model, so as to segment the multiple irregular UI component elements using the SAM model and output the outline map of the multiple irregular UI component elements.

7. An electronic device, characterized in that, The electronic device includes: at least one processor; and at least one memory and bus connected to the processor; The processor and memory communicate with each other via the bus; the processor is used to call program instructions in the memory to execute the UI component analysis method based on the visual large model as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is running, it controls the device where the storage medium is located to execute the UI component analysis method based on a large visual model as described in any one of claims 1 to 5.

Images