UI element parsing method, system, terminal and medium of human-computer interaction interface

Abstract

The application provides a UI element analysis method and system of a human-computer interaction interface, performs semantic region segmentation on a human-computer interaction interface image, and performs fine processing on the result of the semantic region segmentation to obtain block structure information of the human-computer interaction interface; performs positioning and identification of functional UI elements on the human-computer interaction interface image; extracts text information in a specific region of the human-computer interaction interface image; matches the text information and the functional UI elements, and fuses the functional UI elements and the block structure information to analyze the nesting relationship between the functional UI elements to obtain a UI element analysis result of the human-computer interaction interface. The application can efficiently detect, identify and analyze the block structure and UI elements in a complex human-computer interaction interface image in a pure visual scene, significantly improves the operation accuracy and scene generalization ability of intelligent agents such as robot process automation, and further improves the efficiency and reliability of the interaction process automation.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and natural language processing technology, and more specifically, to a method, system, terminal, and medium for parsing UI elements of a human-computer interaction interface. Background Technology

[0002] In the digital age, human-computer interfaces (HCIs) serve as a primary medium for information dissemination, carrying rich text, images, and interactive elements. The combination of computer vision and natural language processing aims to enhance machines' ability to understand HCI content, which is crucial for information extraction, content management, and automated interaction. However, current Robotic Process Automation (RPA) technologies are primarily limited to web-based scenarios and rely on HTML structure rule parsing or HTML-based content parsing. This necessitates the redevelopment of related parsing tools for any HTML structure update, increasing maintenance complexity and cost. Furthermore, for environments where HTML code cannot be directly accessed, such as mobile or desktop applications, existing technologies cannot effectively identify and understand functional UI elements.

[0003] While existing text recognition and image processing technologies can extract and understand human-computer interaction (HCI) content to some extent, they exhibit significant limitations when faced with the diversity and complexity of functional UI elements in HCI scenarios. First, HCI UI elements include various interactive components such as buttons, input boxes, dropdown menus, radio buttons, and checkboxes. These elements not only have varied visual forms but also often carry complex business logic. Existing technologies struggle to accurately identify and parse these elements, especially in dynamic HCI interfaces and complex layouts. Second, although current technical strategies attempt to parse some visual elements when processing HCIs, they often overlook the nesting and hierarchical relationships between elements. This structural information is crucial for accurately mapping and manipulating the interface. Without in-depth analysis of these modular and hierarchical designs, automated tools cannot effectively identify the dependencies and interactions between elements, thus limiting the accuracy and efficiency of operations. Finally, while the comprehensive utilization of multimodal information makes understanding complex interfaces possible, existing parsing technologies typically process visual and text data independently, failing to achieve effective fusion of these two types of information. This has led to existing systems often failing to correctly understand the function and context of UI elements with complex visual and textual content, such as icons with labels or buttons with specific actions, affecting the execution effectiveness of automated tasks and the level of intelligence in processes.

[0004] With the development of Robotic Process Automation (RPA) technology, intelligent parsing of UI elements in general human-computer interaction interfaces has become particularly important. In automated business processes, RPA systems need to accurately identify and manipulate various functional UI elements in the human-computer interaction interface to perform operations such as data entry, information retrieval, and online transactions. This requires not only high accuracy to prevent errors in the business process but also sufficient speed and flexibility to adapt to constantly changing human-computer interaction interface designs and updates.

[0005] In the field of human-computer interaction interface element parsing, early traditional methods were mainly aimed at rule parsing based on HTML code in web page scenarios. Although this strategy performed well when dealing with simple, well-written web pages, its efficiency and accuracy were often limited for complex and dynamically generated content. Secondly, with the updating and iteration of web page content, the existing human-computer interaction interface PRA process also needs to be updated accordingly, which to some extent limits the widespread application of traditional methods.

[0006] With the rise of deep learning methods, especially the development of large-scale language models in Natural Language Processing (NLP), more and more research is exploring the application of these models to RPA tasks in web page scenarios. In early deep learning methods, HTML was used as the main input to the model, hoping to achieve a holistic understanding of web pages and complete tasks such as summarizing web page functions and providing navigation. These language model-based methods attempted to grasp the structure and semantics of the web page interface by parsing the HTML source code. However, this method has a fundamental flaw: it relies excessively on the quality of the HTML source code. Since many real-world web pages have poorly written code, and HTML source code is often insufficient to convey rich visual and structural information, this limits the generalization ability and practicality of such methods. Furthermore, these methods remain limited to web page scenarios and cannot be extended to more general human-computer interaction scenarios, such as mobile applications or desktop application systems.

[0007] In recent years, an increasing number of studies have attempted to address the problem of understanding human-computer interaction (HCI) content by combining visual models with linguistic information in a multimodal model framework. This marks a technological shift from unimodal to multimodal understanding, bringing new possibilities for the parsing of HCI elements in visual scenarios. However, existing technologies still face limitations in HCI updates and complexity management, namely, failing to achieve robust UI element parsing for general HCIs in a single visual scenario. Furthermore, current multimodal understanding technologies in web scenarios require input from web images and DOM information, but DOM information is unavailable in visual scenarios. Additionally, existing multimodal understanding technologies in web scenarios often rely on large linguistic models to understand DOM information, leading to significant inference latency. Summary of the Invention

[0008] To address the aforementioned shortcomings in the prior art, this invention provides a method, system, terminal, and medium for parsing UI elements in a human-computer interaction interface.

[0009] According to one aspect of the present invention, a method for parsing UI elements of a human-computer interaction interface is provided, comprising:

[0010] Semantic region segmentation is performed on the human-computer interaction interface image, and the result of the semantic region segmentation is refined to obtain the block structure information of the human-computer interaction interface.

[0011] Locating and recognizing functional UI elements in human-computer interaction interface images;

[0012] Extract text information from a specific area in a human-computer interaction interface image;

[0013] The text information is matched with the functional UI elements, and the functional UI elements are merged with the block structure information. The nesting relationship between the functional UI elements is analyzed to obtain the UI element parsing result of the human-computer interaction interface.

[0014] Preferably, the step of performing semantic region segmentation on the human-computer interaction interface image and refining the result of the semantic region segmentation to obtain the block structure information of the human-computer interaction interface includes:

[0015] Provide a pre-trained general visual segmentation model, which includes an image encoder and a mask decoder;

[0016] In the image encoder section of the general visual segmentation model, a comprehensive human-computer interaction interface segmentation adaptive fine-tuning module is provided;

[0017] Freeze the image encoder parameters of the general visual segmentation model, and train the general visual segmentation model using the parameters in the integrated human-computer interaction interface segmentation adaptive fine-tuning module;

[0018] The structure of the mask decoder part of the general visual segmentation model is fine-tuned to adapt to the image segmentation results obtained from the integrated human-computer interaction interface segmentation adaptive fine-tuning module, thereby obtaining a semantic region segmentation model;

[0019] The semantic region segmentation model is used to perform semantic region segmentation on human-computer interaction interface images.

[0020] A post-processing module is provided to refine the result of the semantic region segmentation to obtain the block structure information of the human-computer interaction interface. The block structure information is a structured block containing a set of logically related UI elements.

[0021] Preferably, the integrated human-computer interaction interface segmentation adaptive fine-tuning module includes: an image block fine-tuning module, a structural feature fine-tuning module, and an adapter module; wherein:

[0022] The image segmentation fine-tuning module embeds the human-computer interaction interface image from a high-dimensional space into a reduced-dimensional feature space through an adjustable linear layer, which is used to optimize the alignment between the pre-trained model and the target dataset.

[0023] The structural feature fine-tuning module preprocesses the input human-computer interaction interface image and converts it into a grayscale image. It uses the Sobel operator to calculate the gradient magnitude to identify edge and structural features in the human-computer interaction interface image. The obtained edge and structural features are segmented into non-overlapping image blocks consistent with the general visual segmentation model. Adjustable linear layers are used to project these image blocks into a reduced-dimensional feature space to optimize the structural representation of the human-computer interaction interface image.

[0024] The adapter module, combining the outputs of the image block fine-tuning module and the structural feature fine-tuning module, uses a lightweight multilayer perceptron to perform feature enhancement for a specific webpage domain, thereby obtaining image segmentation results.

[0025] Preferably, the post-processing module includes: a spatial context collaborative segmentation and fusion module and an OCR-driven text boundary refinement module, wherein:

[0026] The spatial context collaborative segmentation and fusion module is used to merge overlapping segmentation boxes in the visual segmentation results output by the refined general visual segmentation model.

[0027] The OCR-driven text boundary refinement module uses text data extracted from OCR technology to refine and correct the position of the segmentation box.

[0028] Preferably, the locating and identifying of functional UI elements in the human-computer interaction interface image includes:

[0029] Provide a pre-trained general vision detection model;

[0030] A dataset of UI elements in a human-computer interaction scenario is provided. The general visual detection model is optimized and trained using this dataset to obtain a UI element detection model in a human-computer interaction scenario.

[0031] The human-computer interaction interface image is used as the input of the UI element detection model, and the output is the location and recognition results of functional UI elements in the human-computer interaction interface image.

[0032] Preferably, the extraction of text information within a specific region of the human-computer interaction interface image includes:

[0033] Provide a pre-trained OCR text detection model for obtaining complete webpage text information;

[0034] Based on the errors and application requirements obtained from the actual webpage text detection process using an expert system, the complete webpage text information is filtered to obtain text information within a specific area of ​​the human-computer interaction interface image.

[0035] Preferably, the step of matching the text information with the functional UI elements, fusing the functional UI elements with the block structure information, and parsing the nesting relationships between the functional UI elements to obtain the UI element parsing result of the human-computer interaction interface includes:

[0036] Based on the prior distribution of UI element text information, a UI text matching algorithm is constructed, and the UI text matching algorithm is used to accurately match the functional UI elements with their corresponding text information.

[0037] Based on the block structure information of the human-computer interaction interface, a UI block fusion algorithm is constructed. The functional UI elements are fused with the block structure information using the UI block fusion algorithm to obtain the UI nesting description of the human-computer interaction interface and the UI element parsing result of the human-computer interaction interface.

[0038] Preferably, the UI text matching algorithm includes:

[0039] Obtain prior information on the distribution of text information of UI elements. Based on this prior information, construct a relative position description, which is used to describe the distribution pattern D of different functional UI elements and text information.

[0040]

[0041] In the formula, (x UI ,y UI ) and (x text ,y text ) respectively describe the center point coordinates of the UI element and the text information, (w UI ,h) and (w text ,h text Describe the width and height of UI elements and text information respectively; for different functional UI elements, statistically analyze the distribution of the above distribution pattern D, and obtain the distribution mean. and variance var, As the prior range of text information distribution for the corresponding UI elements;

[0042] Calculate the proportion of text information area of ​​UI elements. Iterate through the detection results of each UI element and calculate the proportion K of the area of ​​the overlapping area between the UI element and the text information to the total text information area.

[0043]

[0044] In the formula, Union(UI,text) refers to the area of ​​the overlapping region between the UI element and the text information, S. text This represents the area of ​​the text information; when K is greater than a set threshold, the text information is used as the text description of the corresponding UI element.

[0045] Let T be the set of text information descriptions that each UI element can obtain. For a UI element with a single matching text information, if len(T) ≥ 1, then only the text information description with the largest K is taken; if len(T) = 0, then based on the prior range of the obtained text information distribution, the text information with the closest relative distance to the UI element is taken as its text information description; for UI elements with multiple matching text information, calculate the maximum area M formed by the UI element and the text information outside each set T:

[0046]

[0047] Take the text information with the smallest M as the additional description of the text information of the UI element, and add it to the set T;

[0048] For the obtained set T of text information descriptions corresponding to UI elements, sort the distances between the coordinates of the text information descriptions in set T and the origin (0,0) of the image coordinates in ascending order to obtain an ordered set. As the final collection of text descriptions for UI elements;

[0049] For the final set of text descriptions obtained Deviation correction and overlap correction are performed separately, and the results are used as the text information description of the corresponding UI element to complete the accurate matching between the UI element and its corresponding text information.

[0050] Preferably, the UI block fusion algorithm includes:

[0051] Initialize a list of potentially nested block structure information indices for each UI element;

[0052] By utilizing the spatial relationship between bounding boxes and block bounding boxes, the block structure information index list is recursively traversed to optimize the matching relationship between UI elements and block structure information;

[0053] The matching relationship between the UI elements and the block structure information is filtered based on the set matching strategy;

[0054] The matching relationship between the filtered UI elements and the block structure information is optimized to ensure that each UI element can be parsed in the logically correct block structure information. For UI elements that fail to match in the block structure information, the UI element is associated with other recent block structure information to ensure that each UI element is in the most appropriate position, thereby completing the fusion between UI elements and block structure information and obtaining the UI nesting description of the human-computer interaction interface.

[0055] According to another aspect of the present invention, a UI element parsing system for a human-computer interaction interface is provided, comprising:

[0056] The block segmentation module is used to perform semantic region segmentation on the human-computer interaction interface image and to refine the result of the semantic region segmentation to obtain the block structure information of the human-computer interaction interface.

[0057] The UI detection module is used to locate and identify functional UI elements in human-computer interaction interface images.

[0058] The text recognition module is used to extract text information from specific areas in human-computer interaction interface images;

[0059] The fusion parsing module is used to match the text information with the UI elements, fuse the UI elements with the block structure information, parse the nesting relationship between the functional UI elements, and obtain the UI element parsing result of the human-computer interaction interface.

[0060] According to a third aspect of the present invention, a computer terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can be used to perform the method described in any one of the above inventions, or to run the system described in the above inventions.

[0061] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can be used to perform the method described in any one of the above-described inventions, or to run the system described in the above-described inventions.

[0062] By adopting the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art:

[0063] The present invention provides a method, system, terminal, and medium for parsing UI elements of human-computer interaction interfaces. In a purely visual scenario, it offers an intelligent parsing technology for UI elements of human-computer interaction interfaces that does not rely on HTML code. This technology can not only effectively understand the visual information in the human-computer interaction interface even when HTML information is missing, but also capture the text information in the human-computer interaction interface. This enables the model to process the content of the human-computer interaction interface more directly and effectively, providing new possibilities for parsing human-computer interaction interface elements, promoting the advancement of human-computer interaction interface processing technology, and providing a more powerful and flexible tool for automating human-computer interaction interface scenarios.

[0064] The present invention provides a method, system, terminal and medium for parsing UI elements of human-computer interaction interfaces. Without relying on the HTML information of the web page scene, it directly understands the UI elements and their structure through a visual model, realizing intelligent parsing of UI elements in a pure visual web page scene.

[0065] The present invention provides a method, system, terminal and medium for parsing UI elements of human-computer interaction interfaces. By combining a UI detection model and a text extraction model, it extends the traditional RPA method based on HTML rule parsing from web page scenarios to general human-computer interaction scenarios. In general human-computer interaction scenarios, such as mobile applications, desktop applications or web page scenarios, it can realize intelligent parsing of UI elements, effectively improving the generalization of human-computer interaction interface RPA.

[0066] The present invention provides a method, system, terminal and medium for parsing UI elements of human-computer interaction interfaces. Through a precise block segmentation algorithm, it provides the nesting relationship of functional UI elements, effectively identifies and parses functional UI elements in human-computer interaction interface scenarios, and improves the efficiency and accuracy of automated tools in processing human-computer interaction interface content in actual operation, especially in dynamic and changing human-computer interaction interface environments.

[0067] The UI element parsing method, system, terminal, and medium provided by this invention significantly improve the accuracy and adaptability of UI element parsing by combining the intuitive perception of computer vision with the deep semantic understanding of natural language processing. This enables the UI to flexibly respond to various changes in UI design and reduce errors and improve efficiency in RPA tasks.

[0068] The UI element parsing method, system, terminal, and medium for human-computer interaction interfaces provided by this invention can be further expanded to include further reasoning on the intelligent parsing results of UI elements in intelligent parsing human-computer interaction scenarios, and prediction of the human-computer interaction interface process, thereby completing the realization of a fully automated parsing and reasoning model for PRA processes in general human-computer interaction scenarios. Attached Figure Description

[0069] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0070] Figure 1 This is a flowchart illustrating the UI element parsing method for a human-computer interaction interface in a preferred embodiment of the present invention.

[0071] Figure 2 This is a flowchart illustrating the process of segmenting the human-computer interaction interface into blocks in a preferred embodiment of the present invention.

[0072] Figure 3 This is a flowchart of the fusion construction module in a preferred embodiment of the present invention.

[0073] Figure 4 This is a schematic diagram of the components of the UI element parsing system for a human-computer interaction interface in a preferred embodiment of the present invention. Detailed Implementation

[0074] The embodiments of the present invention are described in detail below: These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

[0075] To address the limitations of existing Robotic Process Automation (RPA) technologies in managing the complexity of human-computer interaction interfaces (HCIs), this invention provides a method for parsing UI elements of HCIs. This method is used to parse UI elements of HCIs and is applicable to RPA operations in visual scenarios. It can automatically understand and parse functional UI elements and their associated text content. Through this method, complex HCI UI structures can be efficiently identified and parsed directly from image and text data in a purely visual environment, without relying on HTML code, significantly improving the accuracy and application scope of RPA.

[0076] Specifically, such as Figure 1 As shown, the UI element parsing method for the human-computer interaction interface provided in this embodiment may include the following operations:

[0077] S1, perform semantic region segmentation on the human-computer interaction interface image, and refine the results of semantic region segmentation to obtain the block structure information of the human-computer interaction interface;

[0078] S2, locates and identifies functional UI elements in the human-computer interaction interface image;

[0079] S3, extract text information within a specific area of ​​the human-computer interaction interface image;

[0080] S4 matches text information with functional UI elements, merges functional UI elements with block structure information, parses the nesting relationship between functional UI elements, and obtains the UI element parsing results of the human-computer interaction interface.

[0081] In some preferred embodiments, such as Figure 2 As shown, S1 above performs semantic region segmentation on the human-computer interaction interface image, and refines the results of semantic region segmentation to obtain the block structure information of the human-computer interaction interface. It may further include the following operations:

[0082] S11 provides a pre-trained general visual segmentation model, which includes an image encoder and a mask decoder.

[0083] S12 provides a comprehensive human-computer interaction interface segmentation adaptive fine-tuning module in the image encoder part of the general visual segmentation model;

[0084] S13, Freeze the image encoder parameters of the general visual segmentation model, and train the general visual segmentation model using the parameters in the integrated human-computer interaction interface segmentation adaptive fine-tuning module;

[0085] S14, fine-tune the structure of the mask decoder part of the general visual segmentation model to adapt to the image segmentation results obtained from the integrated human-computer interaction interface segmentation adaptive fine-tuning module, and obtain the semantic region segmentation model;

[0086] S15, semantic region segmentation is performed on human-computer interaction interface images using a semantic region segmentation model;

[0087] S16 provides a post-processing module to refine the results of semantic region segmentation and obtain the block structure information of the human-computer interaction interface. The block structure information is a structured block containing a set of logically related UI elements.

[0088] In some preferred embodiments, S11 above, which provides a pre-trained general visual segmentation model, may further include the following operations:

[0089] A general visual segmentation model is provided. The parameters of the general visual segmentation model are trained using a publicly available training dataset to obtain a pre-trained general visual segmentation model.

[0090] In some preferred embodiments, the above-mentioned S12, which integrates the human-computer interaction interface segmentation adaptive fine-tuning module, may further include: an image block fine-tuning module, a structural feature fine-tuning module, and an adapter module; wherein:

[0091] The image segmentation fine-tuning module embeds human-computer interaction interface images from a high-dimensional space into a reduced-dimensional feature space through adjustable linear layers, which is used to optimize the alignment between the pre-trained model and the target dataset.

[0092] The structural feature fine-tuning module preprocesses the input human-computer interaction interface image and converts it into a grayscale image. It uses the Sobel operator to calculate the gradient magnitude to identify edge and structural features in the human-computer interaction interface image. The obtained edge and structural features are segmented into non-overlapping image blocks consistent with the general visual segmentation model. Adjustable linear layers are used to project these image blocks into a reduced-dimensional feature space to optimize the structural representation of the human-computer interaction interface image.

[0093] The adapter module, combining the outputs of the image block fine-tuning module and the structural feature fine-tuning module, uses a lightweight multilayer perceptron to perform feature enhancement for a specific web page domain, thereby obtaining the image segmentation result.

[0094] In some preferred embodiments, the post-processing module in step S16 above may further include: a spatial context collaborative segmentation and fusion module and an OCR-driven text boundary refinement module, wherein:

[0095] The spatial context collaborative segmentation and fusion module is used to merge overlapping segmentation boxes in the visual segmentation results output by the refined general visual segmentation model;

[0096] The OCR-driven text boundary refinement module uses text data extracted from OCR technology to refine and correct the position of the segmentation boxes.

[0097] In some preferred embodiments, the spatial context collaborative segmentation and fusion module is used to merge overlapping segmentation boxes in the visual segmentation results output by the refined general visual segmentation model, and may further include the following operations:

[0098] S1611, Perform spatial analysis on the detected bounding boxes to calculate the position, size and overlapping area of ​​each bounding box;

[0099] S1612, based on the spatial analysis results, merge overlapping or adjacent bounding boxes into a single bounding box, thus completing the merging of overlapping segmentation boxes in the visual segmentation results.

[0100] In some preferred embodiments, the OCR-driven text boundary refinement module, which uses text data extracted from OCR technology to refine and correct the position of the segmentation box, may further include the following operations:

[0101] S1621, uses OCR technology to recognize text data in human-computer interaction interface images and obtains corresponding text boxes;

[0102] S1622, the DBSCAN algorithm is used to perform cluster analysis on text boxes to identify the regional distribution of text data;

[0103] S1623, Based on the clustering of text boxes and the distribution of text data, adjust the position and size of the segmentation boxes to ensure that the segmentation boxes can accurately cover the relevant text data areas.

[0104] In some preferred embodiments, the above-mentioned S2, which involves locating and recognizing functional UI elements in the segmented image of the human-computer interaction interface region, may further include the following operations:

[0105] S21 provides a pre-trained general vision detection model;

[0106] S22 provides a dataset of UI elements in a human-computer interaction scenario. This dataset is used to optimize and train a general visual detection model to obtain a UI element detection model in a human-computer interaction scenario.

[0107] S23, take the human-computer interaction interface image as input to the UI element detection model, and output the location and recognition results of functional UI elements in the human-computer interaction interface image.

[0108] In some preferred embodiments, S21 above, which provides a pre-trained general visual detection model, may further include the following operations:

[0109] A general visual detection model is provided. The parameters of the general visual detection model are trained using a publicly available training dataset to obtain a pre-trained general visual detection model.

[0110] In some preferred embodiments, the above-described S3, which extracts text information within a specific region of the human-computer interaction interface region segmentation image, may further include the following operations:

[0111] S31 provides a pre-trained OCR text detection model for obtaining complete webpage text information; the model has a strong ability to recognize various text formats and layouts in human-computer interaction interface images, including processing text information with multiple fonts, sizes, colors and languages.

[0112] S32, based on the expert system to obtain errors and application requirements in the actual web page text detection process, filters the complete web page text information.

[0113] In some preferred embodiments, such as Figure 3 As shown, S4 above matches text information with functional UI elements, merges functional UI elements with block structure information, parses the nesting relationships between functional UI elements, and obtains the UI element parsing results of the human-computer interaction interface. It may further include the following operations:

[0114] S41, Based on the prior distribution of UI element text information, construct a UI text matching algorithm, and use the UI text matching algorithm to accurately match functional UI elements with their corresponding text information.

[0115] S42. Based on the block structure information of the human-computer interaction interface, a UI block fusion algorithm is constructed. The UI block fusion algorithm is used to fuse functional UI elements with block structure information to obtain the UI nesting description of the human-computer interaction interface and obtain the UI element parsing result of the human-computer interaction interface.

[0116] In some preferred embodiments, the UI text matching algorithm in step S41 above may further include the following operations:

[0117] S411, Obtain the prior knowledge of the distribution of UI element text information. Based on the prior knowledge of the distribution of UI element text information, construct a relative position description. This relative position description is used to describe the distribution pattern D of different functional UI elements and text information.

[0118]

[0119] In the formula, (x UI ,y UI ) and (x text ,y text ) respectively describe the center point coordinates of the UI element and the text information, (w UI ,h) and (w text ,h text Describe the width and height of UI elements and text information respectively; for different functional UI elements, statistically analyze the distribution of the above distribution pattern D, and obtain the distribution mean. and variance var, As the prior range of text information distribution for the corresponding UI elements;

[0120] S412, calculate the proportion of the text information area of ​​the UI element, iterate through the detection results of each UI element, and calculate the proportion K of the area of ​​the overlapping area between the UI element and the text information to the text information area:

[0121]

[0122] In the formula, Union(UI,text) refers to the area of ​​the overlapping region between the UI element and the text information, S. text This represents the area of ​​the text information; when K is greater than a set threshold, the text information is used as the text description of the corresponding UI element.

[0123] S413, let T be the set of text information descriptions that each UI element can obtain; for a UI element with a single text information match, if len(T)≥1, then only the text information description with the largest K is taken; if len(T)=0, then based on the prior range of the obtained text information distribution, the text information with the closest relative distance to the UI element is taken as its text information description; for a UI element with multiple text information matches, calculate the maximum area M formed by the UI element and the text information outside each set T:

[0124]

[0125] Take the text information with the smallest M as the additional description of the text information of the UI element, and add it to the set T;

[0126] S414, for the obtained set T of text information descriptions corresponding to UI elements, sort the distances between the coordinates of the text information descriptions in set T and the origin (0,0) of the image coordinates in ascending order to obtain an ordered set. As the final collection of text descriptions for UI elements;

[0127] S415, for the obtained final text description set Deviation correction and overlap correction are performed separately, and the results are used as the text information description of the corresponding UI element to complete the accurate matching between the UI element and its corresponding text information.

[0128] In some preferred embodiments, the UI block fusion algorithm in step S42 above may further include the following operations:

[0129] S421, initialize a list of potentially nested block structure information indices for each UI element;

[0130] S422: Utilize the spatial relationship between the bounding box and the block bounding box to recursively traverse the block structure information index list and optimize the matching relationship between UI elements and block structure information.

[0131] S423, Filter the matching relationship between UI elements and block structure information based on the set matching strategy;

[0132] S424 optimizes the matching relationship between the filtered UI elements and the block structure information to ensure that each UI element can be parsed in the logically correct block structure information. This parsing refers to the process of obtaining the category information, nesting information, and text information of the UI element. For UI elements that fail to match in the block structure information, the UI element is associated with other recent block structure information to ensure that each UI element is in the most appropriate position, complete the fusion between UI elements and block structure information, and obtain the UI nesting description of the human-computer interaction interface.

[0133] The UI element parsing method for human-computer interaction interfaces provided in the above embodiments of the present invention combines computer vision technology, enabling the identification and parsing of UI elements in a purely visual environment. This significantly improves the accuracy and efficiency of UI element identification and parsing in automated processes, addressing dynamically changing UI designs and content updates, thereby reducing the complexity of RPA development and maintenance workload caused by UI updates. By achieving more accurate and efficient visual parsing, the UI element parsing method for human-computer interaction interfaces provided in the above embodiments of the present invention can optimize the response speed and operational reliability of automation tools, adapt to changing UI environments, and improve the overall performance and generalization of automated processes.

[0134] The technical solution provided by the above embodiments of the present invention will be further described in detail below with reference to a preferred embodiment.

[0135] like Figure 1 The diagram shown is a flowchart of the UI element parsing method for the human-computer interaction interface in this preferred embodiment, including the following steps:

[0136] S1, Block Segmentation: Using a visual segmentation model and post-processing module, semantic region segmentation is performed on the human-computer interaction interface image to obtain block structure information such as the functional attributes of each region module and the nesting relationship between modules.

[0137] S2, UI Detection, uses a specially built and trained UI element detection model to detect, locate, and identify functional UI elements in human-computer interaction interface images, such as text boxes, buttons, drop-down lists, etc.

[0138] S3, text recognition, is used to extract text information within a specific area from human-computer interaction interface images; including text location coordinates and text description content.

[0139] S4, Fusion Parsing: This module merges text and functional UI elements, and through the nested description between UI elements and block structure information, it accurately parses the nesting relationship between UI elements to obtain the UI element parsing results of the human-computer interaction interface.

[0140] In a preferred embodiment, S1 is implemented by employing, as follows Figure 2 The efficient human-computer interaction interface segmentation and adaptation method shown includes the following steps:

[0141] S1-1 provides a pre-connected visual segmentation model, which includes an image encoder and a mask decoder. It uses a general visual segmentation model with high-resolution input (1024×1024 pixels) as its basic architecture. The pre-trained model parameters are obtained through publicly available training datasets (such as ImageNet-1K) to ensure good basic performance for general vision tasks. Simultaneously, a human-computer interaction interface segmentation dataset is acquired, and the images within it undergo standardization processing, such as adjusting brightness and contrast, and normalizing image size, to ensure the consistency of the input data and the adaptability of the model. This data is then used for fine-tuning the general visual segmentation model.

[0142] S1-2 improves the image encoder part of the visual segmentation model by introducing a comprehensive human-computer interaction interface segmentation adaptive fine-tuning module. This module includes an image block fine-tuning module, a structural feature fine-tuning module, and an adapter module, aiming to improve the model's representation ability and generalization performance for specific human-computer interaction interfaces through fine-tuning and fusion; where:

[0143] The image segmentation fine-tuning module projects the original image embedding (i.e., human-computer interaction interface image embedding) from a high-dimensional space to a reduced-dimensional feature space through an adjustable linear layer, which is used to optimize the alignment between the pre-trained model and the target dataset.

[0144] The structural feature fine-tuning module preprocesses the input human-computer interaction interface image and converts it into a grayscale image. It uses the Sobel operator to calculate the gradient magnitude to identify edge and structural features in the human-computer interaction interface image. The obtained edge and structural features are segmented into non-overlapping image blocks consistent with the general visual segmentation model. Adjustable linear layers are used to project these image blocks into a reduced-dimensional feature space to optimize the structural representation of the human-computer interaction interface image.

[0145] The adapter module, combining the outputs of the image segmentation fine-tuning module and the structural feature fine-tuning module, uses a lightweight multilayer perceptron to perform feature enhancement for specific web page domains.

[0146] S1-3, during the training phase, the model is finely tuned using the integrated human-computer interaction interface segmentation adaptive fine-tuning module. This includes: freezing the original image encoder parameters and fine-tuning the structure of the mask decoder to optimize model performance; specifically, freezing the image encoder parameters of the general visual segmentation model and training the general visual segmentation model using the parameters from the integrated human-computer interaction interface segmentation adaptive fine-tuning module; fine-tuning the structure of the mask decoder part of the general visual segmentation model to adapt to the image segmentation results obtained from the integrated human-computer interaction interface segmentation adaptive fine-tuning module, resulting in a semantic region segmentation model; and using the semantic region segmentation model to perform semantic region segmentation on the human-computer interaction interface image.

[0147] S1-4 introduces a post-processing module. This module does not participate in the model training process but is dedicated to further processing the output obtained from the trained model to improve the accuracy and practicality of the segmentation results. This post-processing module includes: a spatial context-based collaborative segmentation and fusion module and an OCR-driven text boundary refinement module, wherein:

[0148] The spatial context collaborative segmentation and fusion module is specifically designed to handle the collaborative fusion of visual segmentation results. It merges overlapping segmentation boxes in the visual segmentation results output by a refined general visual segmentation model, thereby improving the overall accuracy of the results. More preferably, it includes the following steps:

[0149] Spatial analysis is performed on the detected bounding boxes to calculate the position, size, and overlapping area between each bounding box.

[0150] Based on the spatial analysis results, overlapping or adjacent bounding boxes are merged into a single bounding box, thus completing the merging of overlapping segmentation boxes in the visual segmentation results.

[0151] The OCR-driven text boundary refinement module uses text data extracted from OCR technology to refine and correct the position of the segmentation boxes, thereby improving the accuracy and usability of the overall segmentation results; more preferably, it includes the following steps:

[0152] OCR technology is used to recognize text data in human-computer interaction interface images to obtain corresponding text boxes;

[0153] The DBSCAN algorithm is used to perform cluster analysis on text boxes to identify the regional distribution of text data;

[0154] Based on the clustering of text boxes and the distribution of text data, adjust the position and size of the segmentation boxes to ensure that the segmentation boxes can accurately cover the relevant text data areas.

[0155] In a preferred embodiment, S2 is implemented by the following steps:

[0156] S2-1, the initialization phase, provides a pre-trained general visual detection model. This model uses a high-resolution input (1024×1024 pixels) general visual detection model as its basic architecture. The pre-trained model parameters are obtained through training on a publicly available training dataset to ensure good basic performance for general visual tasks. Simultaneously, for the images in the training dataset, standardization processing is performed, such as adjusting brightness and contrast, and normalizing image size, to ensure the consistency of the input data and the adaptability of the model.

[0157] S2-2, the training phase, involves fine-tuning the visual detection model based on the constructed UI element dataset for human-computer interaction scenarios. This optimizes model performance, resulting in the final UI element detection model for human-computer interaction scenarios. In this step, the UI element dataset for human-computer interaction scenarios is constructed by manually annotating UI elements in the scene images. It can contain over 20,000 images and more than 20 categories of UI elements, enabling intelligent UI parsing for general human-computer interaction scenarios.

[0158] S2-3 takes the human-computer interaction interface image as input to the UI element detection model and outputs the location and recognition results of functional UI elements in the human-computer interaction interface image.

[0159] In a preferred embodiment, S3 is implemented by the following steps:

[0160] S3-1, Text Detection, provides a pre-trained OCR text detection model. It uses a high-resolution input (1024×1024 pixels) OCR text detection model as its basic architecture and adopts the parameters of existing open-source Chinese and English OCR models to ensure that the model has good basic performance for Chinese and English text in order to obtain complete web page text information.

[0161] S3-2, Text Filtering: Based on errors and application requirements discovered during extensive testing of expert systems in the actual webpage text detection process, the complete webpage text information obtained above is filtered.

[0162] In a preferred embodiment, S4 is implemented by employing, as follows Figure 3 The fusion parsing method shown includes the following steps:

[0163] S4-1, UI-Text Matching: Based on prior information about UI elements and their corresponding text in the collected webpage dataset, a specific UI-text matching algorithm is constructed to accurately match UI elements with their corresponding text descriptions. More preferably, for special UI elements, such as input boxes and dropdown lists, multiple text elements can be matched to ensure a sufficient description of the UI element. The specific operation is as follows:

[0164] S4-1-1, Obtaining Prior Information on the Distribution of UI Element Text Information: Based on a pre-collected webpage dataset, prior information on the distribution of functional UI elements and their text descriptions is obtained. The following relative position descriptions are designed to describe the patterns between different functional UI elements and their text descriptions:

[0165]

[0166] In the above formula (x UI ,y UI ) and (x text ,y text ) respectively describe the center point coordinates of the UI element and the text, (w UI ,h) and (w text ,h text Describe the width and height of UI elements and text respectively. For different functional UI elements, calculate the distribution of D above and obtain the distribution mean. and variance var, This serves as the prior range for the text distribution of the corresponding UI elements.

[0167] S4-1-2, calculate the text area ratio of the UI element. Iterate through each UI detection result and calculate the ratio K of the overlapping area between the UI and the text to the text area. The formula is as follows:

[0168]

[0169] In the above formula, Union(UI,text) refers to the area of ​​the overlapping region between UI and text, S text The text area refers to the area of ​​the text; if K > 0.5, then the text is used as the text description of the corresponding UI.

[0170] S4-1-3, UI element text matching, in the above formula, each UI element can obtain a corresponding text description set T. For UI elements that only require a single text, if len(T)≥1, then only the text description with the largest K is taken; if len(T)=0, then based on the prior range of text distribution obtained in S4-1-1, the text with the closest relative distance to the UI element is taken as its text description. For UI elements with multiple text matching, such as input boxes, dropdown lists, etc., the above method often only obtains the corresponding text within the UI element, but cannot obtain the corresponding descriptive text of the UI element, such as "name", etc. Therefore, for these UI elements, the maximum area M formed by the UI element and the text outside each set T is calculated, as follows:

[0171]

[0172] Take the text corresponding to the smallest M as the additional description text for this UI element, and add it to the set T.

[0173] S4-1-4, Text Description Sorting: For the obtained set T of text descriptions corresponding to the UI, calculate the distance of the description coordinates of the text in set T from the image coordinate origin (0,0) and sort them in ascending order to obtain an ordered set. As a collection of text descriptions for the UI.

[0174] S4-1-5, Post-processing of text matching, for the obtained text description set Post-processing optimization will be performed, which mainly includes two steps: (1) bias correction, for the obtained text description set Calculate whether the corresponding text position and UI element have a large pixel-level deviation on the X-axis or Y-axis; for example, the radio button corresponding element and text description are basically consistent on the Y-axis, and relatively close on the X-axis and on the left; if there is a large deviation, the existing text description will be removed, and the matching process from S4-1-2 to S4-1-4 will be repeated. (2) Overlap correction, for the obtained text description set Calculate whether there is a large Intersection over Union (IOU) between the corresponding text location and the UI element: For example, for elements such as buttons, the text description and the UI location highly overlap, resulting in a large IOU. Text outside the UI element is not the descriptive text of the corresponding element. Therefore, in this case, the image of the UI element area is separately processed by an OCR text recognition model, and the result is used as the corresponding UI text description.

[0175] S4-2, UI-Block Fusion: Based on the block structure information of the human-computer interaction interface obtained in step S1, a specific UI-block fusion algorithm is constructed to fuse UI element information and block structure information, obtaining a nested UI description of the human-computer interaction interface, which serves as the intelligent parsing result of the UI elements in the human-computer interaction interface. The specific operation is as follows:

[0176] S4-2-1, UI and Block Initialization Matching. In this sub-step, a list of potentially nested block node indices is first initialized for each UI element. By starting from the root node of the entire interface, this operation establishes a basic relational framework for hierarchical structural analysis of UI elements. This initialization ensures that subsequent precise matching steps can be performed on a clearly defined structure.

[0177] S4-2-2, Recursive Exploration and Matching Refinement. After initial matching, the next step is to refine the matching between the UI and blocks by recursively traversing the block structure of the human-computer interaction interface. During this process, the containment relationship between the UI element's bounding box and the block's bounding box is determined. This recursive exploration not only checks whether each block contains a specific UI element, but also updates the block node information, recording which UI elements are associated with it. The key to this step is accurately placing UI elements within their logically correct blocks, ensuring the accuracy of the UI's structured description.

[0178] S4-2-3, Block Type Selection and Matching Strategy Application. Further, this step considers the impact of different block types on UI element matching, especially large content blocks such as advertisements or background images. Here, different strategies are applied depending on whether UI elements are matched with large content blocks. By adjusting the matching algorithm to include or exclude these blocks, various UI design scenarios can be handled more flexibly, thus better adapting to different interface layout requirements. In this step, the matching strategies specifically include:

[0179] Block type identification: First, identify the interface elements that are large content blocks. These interface elements are usually image-dense or media-dense areas, including large advertising banners, video playback areas, or visually attractive background images.

[0180] Matching criteria settings: Setting criteria determines whether UI elements are matched with these large content blocks. This includes considering the functional requirements of the UI elements and the visual characteristics of the blocks. For example, for UI elements that require user interaction (such as buttons or input fields), avoid matching them with visually distracting background blocks.

[0181] Spatial Relationship Assessment: Analyze the spatial relationship between UI elements and large content blocks to determine whether they should be logically related.

[0182] Matching strategy execution: Implement specific matching strategies to determine under what circumstances UI elements should be associated with large content blocks, or excluded from those blocks. This involves adjusting the bounding boxes of UI elements to better fit their functional blocks, or modifying the properties of the blocks to reduce visual clutter to the UI elements.

[0183] S4-2-4, Optimization of UI and Block Relationships After Matching. This step optimizes the relationship between the UI and blocks obtained after the matching process, including adjusting the relationship between UI elements and blocks to ensure that each UI element can be identified and interpreted in the logically correct block. Particular attention is paid to the handling of similar UI elements. Similar UI elements refer to UI components that are similar in category and size. By analyzing the distribution of these components in specific blocks, they are matched with their corresponding text descriptions to ensure that the function and purpose of each UI component are accurately explained. Furthermore, the matching results are optimized, including adjusting the relationship between UI elements and blocks to ensure that each UI element can be identified and interpreted in the logically correct block. This process not only improves the accuracy of UI parsing but also enhances the quality of user interaction. Finally, for UI elements that fail to match within large content blocks, they are attempted to be associated with the nearest sub-blocks to further optimize the overall interface structure, ensuring that each UI element is in the most appropriate position. Through these steps, not only is the relationship between the UI and blocks optimized, but the functionality and usability of the entire user interface are also improved, ensuring the consistency and efficiency of the interaction design.

[0184] One embodiment of the present invention provides a UI element parsing system for human-computer interaction interfaces.

[0185] Specifically, such as Figure 4 As shown, the UI element parsing system for human-computer interaction provided in this embodiment may include the following modules:

[0186] The block segmentation module is used to perform semantic region segmentation on the human-computer interaction interface image and to refine the results of semantic region segmentation to obtain the block structure information of the human-computer interaction interface.

[0187] The UI detection module is used to locate and identify functional UI elements in human-computer interaction interface images.

[0188] The text recognition module is used to extract text information from specific areas in human-computer interaction interface images;

[0189] The fusion parsing module is used to match text information with UI elements, fuse UI elements with block structure information, parse the nesting relationship between functional UI elements, and obtain the UI element parsing results of the human-computer interaction interface.

[0190] It should be noted that the steps in the method provided by the present invention can be implemented using corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the method to realize the composition of the system. That is, the embodiments in the method can be understood as preferred examples for building the system, and will not be elaborated here.

[0191] An embodiment of the present invention also provides a computer terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can be used to execute the method of any of the above embodiments of the present invention, or to run the system of any of the above embodiments of the present invention.

[0192] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs, functional modules, etc. that implement the above methods), computer instructions, etc., and the aforementioned computer programs, computer instructions, etc., can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0193] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0194] A processor is used to execute computer programs stored in memory to implement the various steps of the methods or various modules of the systems involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method and system embodiments.

[0195] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0196] An embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to perform the method of any of the above embodiments of the present invention, or to run the system of any of the above embodiments of the present invention.

[0197] Those skilled in the art will understand that embodiments of the present invention can be provided as methods or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0198] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0199] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0200] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0201] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0202] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0203] The UI element parsing method, system, terminal, and medium for human-computer interaction interfaces provided in the above embodiments of the present invention belong to a visual scene-based intelligent parsing technology for UI elements of human-computer interaction interfaces, overcoming the limitations of traditional HTML rule parsing of UI elements in this field. By introducing element detection and text extraction, functional UI elements and their text information are extracted from human-computer interaction interface images in a visual scene. By constructing a general functional UI dataset for human-computer interaction interface scenes, the model's ability to detect UI element content in human-computer interaction interface scenes is significantly improved.

[0204] The UI element parsing method, system, terminal, and medium for human-computer interaction interfaces provided in the above embodiments of the present invention employ block segmentation technology to further segment the image content of existing visual human-computer interaction interfaces into structured blocks, providing structured information for constructing the nesting relationship of UI elements within the human-computer interaction interface; through fusion construction technology, the detected functional UI elements and extracted text information are associated with each other through image-text matching algorithms, and the nesting and logical relationships between elements are constructed in combination with the block segmentation results to conform to the cognitive structure of user interaction; the technical solution provided in the above embodiments of the present invention not only successfully detects and identifies functional UI elements in visual scenes of human-computer interaction interfaces, but also provides text information descriptions and nesting relationship descriptions for different UI elements, and makes the model more accurate and efficient in practical application scenarios (such as human-computer interaction interface RPA).

[0205] Any matters not covered in the above embodiments of the present invention are well-known in the art.

[0206] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for parsing UI elements in a human-computer interaction interface, characterized in that, include: Semantic region segmentation is performed on the human-computer interaction interface image, and the result of the semantic region segmentation is refined to obtain the block structure information of the human-computer interaction interface. Locating and recognizing functional UI elements in human-computer interaction interface images; Extract text information from a specific area in a human-computer interaction interface image; The text information is matched with the functional UI elements, and the functional UI elements are merged with the block structure information. The nesting relationship between the functional UI elements is analyzed to obtain the UI element parsing result of the human-computer interaction interface. in: The step of performing semantic region segmentation on the human-computer interaction interface image and refining the results of the semantic region segmentation to obtain the block structure information of the human-computer interaction interface includes: Provide a pre-trained general visual segmentation model, which includes an image encoder and a mask decoder; In the image encoder section of the general visual segmentation model, a comprehensive human-computer interaction interface segmentation adaptive fine-tuning module is provided; Freeze the image encoder parameters of the general visual segmentation model, and train the general visual segmentation model using the parameters in the integrated human-computer interaction interface segmentation adaptive fine-tuning module; The structure of the mask decoder part of the general visual segmentation model is fine-tuned to adapt to the image segmentation results obtained from the integrated human-computer interaction interface segmentation adaptive fine-tuning module, thereby obtaining a semantic region segmentation model; The semantic region segmentation model is used to perform semantic region segmentation on human-computer interaction interface images. A post-processing module is provided to refine the result of the semantic region segmentation to obtain the block structure information of the human-computer interaction interface. The block structure information is a structured block containing a set of logically related UI elements. The UI text matching algorithm includes: Obtain prior information on the distribution of text information of UI elements. Based on this prior information, construct a relative position description, which is used to describe the distribution pattern D of different functional UI elements and text information. ； In the formula, and Describe the center point coordinates of the UI element and the text information respectively. and Describe the width and height of UI elements and text information respectively; for different functional UI elements, statistically analyze the distribution of the above distribution pattern D and obtain the distribution mean. and variance ,Will As the prior range of text information distribution for the corresponding UI elements; Calculate the proportion of text information area for each UI element. Iterate through the detection results of each UI element and calculate the proportion of the overlapping area between the UI element and the text information to the total text information area. : ； In the formula, Refers to the area of ​​overlap between UI elements and text information. This represents the area of ​​the text information; when K is greater than a set threshold, the text information is used as the text description of the corresponding UI element. Let T be the set of text information descriptions that each UI element can obtain; for a UI element that matches a single piece of text information, if Then only the text information description corresponding to the largest K is taken; if Based on the prior range of the obtained text information distribution, the text information closest to the UI element is taken as its text information description; for multiple text information matching UI elements, the maximum area M formed by the UI element and the text information outside each set T is calculated: ； Take the text information with the smallest M as the additional description of the text information of the UI element, and add it to the set T; For the obtained set T of text information descriptions corresponding to UI elements, the coordinates of the text information descriptions in set T are compared with the origin of the image coordinates. Sort the distances between them in ascending order to obtain an ordered set. , which serves as the final text description set for UI elements; For the final set of text descriptions obtained Deviation correction and overlap correction are performed respectively, and the results are used as the text information description of the corresponding UI element to complete the accurate matching between the UI element and its corresponding text information. The UI block fusion algorithm includes: Initialize a list of potentially nested block structure information indices for each UI element; By utilizing the spatial relationship between bounding boxes and block bounding boxes, the block structure information index list is recursively traversed to optimize the matching relationship between UI elements and block structure information; The matching relationship between the UI elements and the block structure information is filtered based on the set matching strategy; The matching relationship between the filtered UI elements and the block structure information is optimized to ensure that each UI element can be parsed in the logically correct block structure information. For UI elements that fail to match in the block structure information, the UI element is associated with other recent block structure information to ensure that each UI element is in the most appropriate position, thereby completing the fusion between UI elements and block structure information and obtaining the UI nesting description of the human-computer interaction interface.

2. The method for parsing UI elements of a human-computer interaction interface according to claim 1, characterized in that, It also includes any one or more of the following: - The integrated human-computer interaction interface segmentation adaptive fine-tuning module includes: an image block fine-tuning module, a structural feature fine-tuning module, and an adapter module; wherein: The image segmentation fine-tuning module embeds the human-computer interaction interface image from a high-dimensional space into a reduced-dimensional feature space through an adjustable linear layer, which is used to optimize the alignment between the pre-trained model and the target dataset. The structural feature fine-tuning module preprocesses the input human-computer interaction interface image and converts it into a grayscale image. It uses the Sobel operator to calculate the gradient magnitude to identify edge and structural features in the human-computer interaction interface image. The obtained edge and structural features are segmented into non-overlapping image blocks consistent with the general visual segmentation model. Adjustable linear layers are used to project these image blocks into a reduced-dimensional feature space to optimize the structural representation of the human-computer interaction interface image. The adapter module, combining the outputs of the image block fine-tuning module and the structural feature fine-tuning module, uses a lightweight multilayer perceptron to perform feature enhancement for a specific web page domain, thereby obtaining image segmentation results; - The post-processing module includes: a spatial context collaborative segmentation and fusion module and an OCR-driven text boundary refinement module, wherein: The spatial context collaborative segmentation and fusion module is used to merge overlapping segmentation boxes in the visual segmentation results output by the refined general visual segmentation model. The OCR-driven text boundary refinement module uses text data extracted from OCR technology to refine and correct the position of the segmentation box.

3. The UI element parsing method for human-computer interaction interfaces according to claim 1, characterized in that, The process of locating and recognizing functional UI elements in human-computer interaction interface images includes: Provide a pre-trained general vision detection model; A dataset of UI elements in a human-computer interaction scenario is provided. The general visual detection model is optimized and trained using this dataset to obtain a UI element detection model in a human-computer interaction scenario. The human-computer interaction interface image is used as the input of the UI element detection model, and the output is the location and recognition results of functional UI elements in the human-computer interaction interface image.

4. The method for parsing UI elements of a human-computer interaction interface according to claim 1, characterized in that, The extraction of text information within a specific region of the human-computer interaction interface image includes: Provide a pre-trained OCR text detection model for obtaining complete webpage text information; Based on the errors and application requirements obtained from the actual webpage text detection process using an expert system, the complete webpage text information is filtered to obtain text information within a specific area of ​​the human-computer interaction interface image.

5. The method for parsing UI elements of a human-computer interaction interface according to claim 1, characterized in that, The process of matching the text information with the functional UI elements, fusing the functional UI elements with the block structure information, and parsing the nesting relationships between the functional UI elements to obtain the UI element parsing results of the human-computer interaction interface includes: Based on the prior distribution of UI element text information, a UI text matching algorithm is constructed, and the functional UI element is accurately matched with its corresponding text information using the UI text matching algorithm. Based on the block structure information of the human-computer interaction interface, a UI block fusion algorithm is constructed. The functional UI elements are fused with the block structure information using the UI block fusion algorithm to obtain the UI nesting description of the human-computer interaction interface and obtain the UI element parsing result of the human-computer interaction interface.

6. A UI element parsing system for implementing the method of any one of claims 1-5, characterized in that, include: The block segmentation module is used to perform semantic region segmentation on the human-computer interaction interface image and to refine the result of the semantic region segmentation to obtain the block structure information of the human-computer interaction interface. The UI detection module is used to locate and identify functional UI elements in human-computer interaction interface images. The text recognition module is used to extract text information from specific areas in human-computer interaction interface images; The fusion parsing module is used to match the text information with the UI elements, fuse the UI elements with the block structure information, parse the nesting relationship between the functional UI elements, and obtain the UI element parsing result of the human-computer interaction interface.

7. A computer terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it can be used to perform the method of any one of claims 1-5, or to run the system of claim 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program can be used to perform the method of any one of claims 1-5, or to run the system of claim 6.

Images