Graphical user interface instruction positioning method and system based on dynamic area search
By using a dynamic region search method, combined with semantic relevance and interface coverage consistency assessment, the observation range is dynamically adjusted, which solves the problem of inaccurate target localization in complex interfaces and achieves efficient and stable interface element localization, which is suitable for multimodal large models.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANKAI UNIV
- Filing Date
- 2026-04-14
- Publication Date
- 2026-06-26
AI Technical Summary
Existing graphical user interface positioning methods struggle to accurately locate target interface elements corresponding to user commands in complex interfaces with high resolution, dense elements, and a large amount of irrelevant interference information. They are easily affected by irrelevant areas, and unidirectional scaling strategies are unable to recover errors from early deviations.
A dynamic region search-based approach is adopted. By receiving natural language instructions and interface screenshots, candidate region elements are parsed, semantic relevance scores and interface coverage consistency are calculated, focusing, shifting or expanding perception actions are performed, a region search tree is constructed, the optimal search path is selected, and finally the target element is located in a multimodal large model.
It improves the accuracy and stability of target localization in complex interfaces, reduces interference from irrelevant areas, avoids error accumulation, and is applicable to existing multimodal large models to improve performance without additional training. It has versatility and scalability.
Smart Images
Figure CN122019045B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image or video recognition technology, and in particular to a graphical user interface command localization method and system based on dynamic region search. Background Technology
[0002] The task of command localization in a graphical user interface (GUI) can typically be described as follows: given a screenshot of the current interface and a natural language command, the system needs to accurately locate the target interface element on the screen that semantically corresponds to the command, such as a button, icon, menu item, input box, or other interactive control, to provide a reliable spatial location basis for subsequent actions. This task is the fundamental step for the GUI agent to understand user intent and complete interactive operations, and its accuracy directly affects the reliability of subsequent clicks, inputs, and other operations.
[0003] Existing graphical user interface (GUI) localization methods can be mainly divided into two categories. The first category adopts a full-screen, single-step prediction paradigm, which directly regresses the target coordinates or predicts the target bounding box on the entire interface image based on the complete screenshot and user commands. This type of method is relatively straightforward, but because real interfaces are usually high-resolution, have a large number of elements, are densely arranged, and contain a lot of text, icons, and controls unrelated to the current task, the model is easily disturbed by irrelevant areas in the global view, leading to distraction and affecting the accuracy and stability of target localization. The second category of methods uses multi-step cropping or progressive scaling to first roughly locate candidate regions and then continuously narrow the observation range to gradually approach the target control. This type of method can improve the local fine-grained perception ability to a certain extent, but its search path is usually unidirectionally contracting. Once the early cropped area deviates from the real target, it is often difficult to recover in subsequent processes, and errors are prone to gradual accumulation. Especially in complex interfaces with high resolution, dense interface elements, and high local semantic similarity, simply relying on the forward scaling strategy is difficult to reliably handle situations such as view offset, context loss, and confusion between the target and distractors.
[0004] Furthermore, real-world interfaces differ from typical natural images, often exhibiting significant structural heterogeneity. On one hand, interfaces contain textual elements, icon elements, combo controls, list items, menu bars, and multi-layered nested containers; on the other hand, the visual salience of target elements is not always prominent. Many controls are small in size, their icons look similar, and their functional semantics require consideration of surrounding text or layout context for accurate judgment. Therefore, relying solely on raw visual features or a single global match is usually insufficient to reliably extract truly relevant regional cues from complex interfaces. Summary of the Invention
[0005] The technical problem to be solved by the present invention is to provide a graphical user interface command localization method and system based on dynamic region search, which can more accurately locate the target interface element corresponding to the user command in a complex interface with high resolution, dense elements and a large amount of irrelevant interference information without additional training.
[0006] This invention is achieved through the following technical solution:
[0007] A graphical user interface instruction location method based on dynamic region search includes the following steps:
[0008] S1: Receive natural language commands input by the user and obtain a screenshot of the current complete interface;
[0009] S2: Parse the elements of the candidate area in the current complete interface screenshot, output a structured element set containing bounding boxes, semantic descriptions and interactive attributes, and combine the natural language instructions input by the user. Based on the semantic description, calculate the semantic relevance score between the element and the instruction for each element in the structured element set. Based on the bounding box, calculate the interface coverage consistency within the candidate area. Based on the semantic relevance score and interactive attributes, calculate the element relevance score and semantic concentration.
[0010] S3: Based on the structured element set and the semantic relevance scores of each element and the instruction, perform one of the three types of perception actions: focusing, shifting, or expanding, to generate a new candidate region, and calculate the semantic relevance scores of each element and the instruction in the new candidate region, the interface coverage consistency, the element relevance score, and the semantic concentration in the new candidate region.
[0011] S4: Quality scores are given to all candidate regions based on the relevance score of elements in the candidate region or new candidate region, the consistency of interface coverage within the candidate region or new candidate region, and the semantic concentration.
[0012] S5: Plan and search for candidate regions generated by different sensory actions. Based on the current candidate region status, candidate actions, newly generated candidate regions, and quality scores of all candidate regions, construct a region search tree, select the optimal search path within a given search budget, and output the best region.
[0013] S6: Input the optimal region and natural language instructions into the basic multimodal large model, and output the location prediction results of the target element.
[0014] Furthermore, in steps S2 and S3, the semantic relevance score between each element and the instruction is calculated according to equation (1):
[0015] (1);
[0016] in: Indicates the first The semantic relevance score of each element to the current instruction. Indicates user commands, Indicates matrix transpose. Indicates a text encoder. Indicates the first Each element constructs the corresponding text description. This represents the L2 norm.
[0017] Furthermore, in steps S2 and S3, the interface coverage consistency within the current candidate region is calculated according to equation (2):
[0018] (2);
[0019] in: This indicates that the interface coverage within the current candidate region is consistent. Indicates the current candidate region. Indicates area, Indicates the first candidate in the current candidate region The bounding box coordinates of each element. This indicates the total number of elements in the current candidate region.
[0020] Furthermore, in steps S2 and S3, the element relevance score is calculated according to equation (3), and the semantic concentration is calculated according to equation (4):
[0021] (3);
[0022] (4);
[0023] in: This represents the relevance score of the elements in the current candidate region. Indicates the first candidate in the current candidate region Each element interacts with and perceives weight. This represents a constant that avoids a denominator of zero. This indicates the semantic concentration of elements in the current candidate region. Indicates the first candidate in the current candidate region The percentage of the semantic relevance score of each element to the current instruction in the total semantic relevance scores of all elements to the current instruction. This represents the natural exponential function. Indicates temperature parameter, This represents the semantic relevance score between any element in the current set of structured elements and the current instruction.
[0024] Furthermore, in step S3, based on the structured element set and the semantic relevance score of each element to the instruction, one of three perceptual actions—focusing, shifting, or expanding—is performed to generate a new candidate region:
[0025] S311: Compare the number of elements in the current candidate region with the number of elements in the previous round candidate region. If the number of elements in the current candidate region is greater than or equal to 90% of the number of elements in the previous round candidate region, then perform a focusing action, select elements from the current candidate region that have a higher semantic relevance score to the instruction, calculate the minimum enclosing region of the selected elements, and use the minimum enclosing region of the selected elements as a new candidate region. If the number of elements in the current candidate region is less than 90% of the number of elements in the previous round candidate region, then proceed to the next step.
[0026] S312: Find elements outside the current candidate region that have a high semantic relevance score to the instruction, perform a transition action, calculate the minimum enclosing region of the selected element, and use the minimum enclosing region as a new candidate region, or perform an expansion action to merge the selected element with the elements in the current candidate region to form a new candidate region.
[0027] Furthermore, in step S4, a quality score is calculated for all candidate regions according to equation (5):
[0028] (5);
[0029] in: This indicates the quality score of the candidate region. This indicates the relative contribution of the element's correlation score. This represents the relative contribution to the consistency of interface coverage within the current candidate region. This represents the relative contribution of the semantic concentration of the elements in the current candidate region.
[0030] In the optimized step S5, a Monte Carlo tree search strategy is used to plan and search for candidate regions generated by different perceptual actions.
[0031] The optimized current candidate region state in step S5 includes the current candidate region, the structured element set of the current candidate region, and the semantic relevance score of each element in the structured element set of the current candidate region to the instruction.
[0032] Furthermore, in step S5, the method for constructing a region search tree and selecting the optimal search path within a given search budget to output the best region is as follows:
[0033] S511: Each node in the search tree represents a candidate region state, and the edges in the search tree represent perceived actions. Starting from the root node, the most promising action branch is selected according to the Monte Carlo tree search strategy.
[0034] S512: After reaching the leaf node, generate new candidate regions for the perception actions that have not yet been expanded, and add them to the search tree;
[0035] S513: Use the quality score of the candidate region as the reward of the leaf node, and backpropagate the reward of the leaf node along the search path to update the statistics of each node in the search tree.
[0036] S514: After multiple rounds of searching, select the candidate region with the highest quality score or the most visits from the search tree as the best region.
[0037] A graphical user interface instruction localization system based on dynamic region search is used to execute a graphical user interface instruction localization method based on dynamic region search as described in any of the above, comprising a task input module, an interface acquisition module, an element perception module, a dynamic perception action module, a region quality assessment module, an action planning module, and a final localization module.
[0038] The task input module is used to receive natural language instructions input by the user;
[0039] The interface acquisition module is used to acquire a screenshot of the current complete interface.
[0040] The element perception module is used to parse the elements in the candidate area of the current complete interface screenshot, output a structured element set, and calculate the semantic relevance score between the elements and the instructions, the interface coverage consistency within the candidate area, the element relevance score, and the semantic concentration.
[0041] The dynamic perception action module is used to generate new candidate regions by performing one of three types of perception actions—focusing, shifting, or expanding—based on the structured element set and the semantic relevance score between each element and the instruction.
[0042] The regional quality assessment module scores all candidate regions based on element correlation scores, interface coverage consistency within candidate regions, and semantic concentration.
[0043] The action planning module is used to plan and search for candidate regions generated by different perceptual actions. Based on the current candidate region status, candidate actions, newly generated candidate regions, and quality scores of all candidate regions, it constructs a region search tree, selects the optimal search path within a given search budget, and outputs the best region.
[0044] The final localization module is used to input the optimal region and natural language instructions into the basic multimodal large model and output the location prediction results of the target element.
[0045] Beneficial effects of the invention:
[0046] This invention proposes a dynamic region search method that requires no additional training and can be seamlessly integrated into existing multimodal large models. It can more accurately locate target interface elements corresponding to user commands in high-resolution, element-dense, and highly irrelevant graphical user interfaces (GUIs) containing a large amount of irrelevant interference. Compared to methods that perform single-step prediction across the entire screen, this invention effectively reduces the interference of irrelevant regions on model judgments by first searching for high-quality regions and then performing final localization, thus improving the stability and reliability of the localization results. Simultaneously, this invention introduces three perceptual actions: focusing, shifting, and expanding. These actions dynamically adjust the observation range based on the strength of cues in the current candidate region, avoiding the problem of difficulty in recovering from early deviations in fixed-path step-by-step pruning methods. Furthermore, this invention uses a region quality assessment mechanism to comprehensively evaluate the correlation between candidate regions and user commands, the coverage of interface elements within the region, and the degree of semantic concentration. It also combines this with a planned search strategy to select different search paths, thereby improving the target localization capability in complex interface scenarios. In addition, this invention adopts a training-independent plug-and-play design, which improves the performance of existing models in GUI command localization tasks without requiring retraining or fine-tuning of the base model. It possesses good versatility, scalability, and engineering application value. Attached Figure Description
[0047] Figure 1 This is a schematic diagram of the process of this invention. Detailed Implementation
[0048] A graphical user interface command location method based on dynamic region search is illustrated in the flowchart below. Figure 1 As shown, it includes the following steps:
[0049] S1: Receive natural language commands input by the user and obtain a screenshot of the current complete interface;
[0050] The task input module receives natural language commands from the user, outputs the task text, and sends it to the interface element perception module and the final localization module as input for semantic matching and final localization. The interface input module captures a complete screenshot of the current interface, outputs the original complete interface image, and sends it to the interface element perception module as input for element parsing, while also using it as the initial search range in the dynamic region search process.
[0051] S2: Parse the elements of the candidate area in the current complete interface screenshot, output a structured element set containing bounding boxes, semantic descriptions and interactive attributes, and combine the natural language instructions input by the user. Based on the semantic description, calculate the semantic relevance score between the element and the instruction for each element in the structured element set. Based on the bounding box, calculate the interface coverage consistency within the candidate area. Based on the semantic relevance score and interactive attributes, calculate the element relevance score and semantic concentration.
[0052] This step is completed by the interface element perception module, which outputs the structured element set and calculated indicators to the subsequent dynamic perception action module and action planning module.
[0053] Specifically, the interface element perception module uses an interface element parser to parse the elements in the candidate area of the current complete interface screenshot, extracting visible UI elements and forming a structured record for each element. For each parsed element, its bounding box position in the interface is recorded. If the element contains text, the identified text is used as the semantic description of the element; if the element does not contain explicit text but is an icon-type element, its semantic description is generated using icon description. Simultaneously, it records whether the element is interactive for subsequent candidate area quality evaluation. To improve semantic matching, user commands and element semantic descriptions can be further unified with semantic encoding. Specifically, based on the application type and system platform information of the current interface, a unified domain prefix can be added to user commands and element semantic descriptions, placing them in the same interface semantic space. Subsequently, a text embedding model is used to encode user commands and element semantic descriptions respectively, and the semantic relevance score between each element and the command is calculated.
[0054] Specifically, the extracted element bounding box is represented as The set of elements covered by the current candidate region is ,in: For the first Each element's bounding box, Indicates the first The bounding box coordinates of each element. Indicates the first Each element constructs the corresponding text description. Indicates the first Element categories, Indicates the first The depth of an element in the hierarchy. Indicates the first Whether an element is clickable or focusable. This represents the set of elements covered by the candidate region. Indicates the current candidate region. express With the current candidate region coverage ratio, , Indicates area, Indicates intersection, This represents a constant that avoids a denominator of zero. This represents the threshold for the coverage ratio of an element to the current candidate region.
[0055] After obtaining the structured set of elements, the system needs to determine which elements are more likely to be related to the current user command. To do this, a corresponding text description is constructed for each element, which can consist of the element text, icon semantics, and necessary interface context information. A text encoder can be used to encode both the user command and the element's text description separately, and cosine similarity can be used to calculate the semantic relevance score between each element and the command.
[0056] S3: Based on the structured element set and the semantic relevance scores of each element and the instruction, perform one of the three types of perception actions: focusing, shifting, or expanding, to generate a new candidate region, and calculate the semantic relevance scores of each element and the instruction in the new candidate region, the interface coverage consistency, the element relevance score, and the semantic concentration in the new candidate region.
[0057] Specifically, in steps S2 and S3, the semantic relevance score between each element and the instruction can be calculated according to equation (1), the interface coverage consistency within the current candidate region can be calculated according to equation (2), the element relevance score can be calculated according to equation (3), and the semantic concentration can be calculated according to equation (4).
[0058] (1);
[0059] in: Indicates the first The semantic relevance score of each element to the current instruction. Indicates user commands, Indicates matrix transpose. Indicates a text encoder. Indicates the first Each element constructs the corresponding text description. This represents the L2 norm.
[0060] This indicates the first The degree of semantic matching between an element and the current instruction; the higher the score, the more likely the element is to be related to the target control.
[0061] (2);
[0062] in: This indicates the consistency of interface coverage within the current candidate area. Interface coverage consistency measures whether the current candidate area contains sufficient real interface structure, rather than large areas of white space, background areas, or irrelevant decorative areas. Indicates the first candidate region The area of each element, This represents the total area of the candidate region. This indicates the total number of elements in the current candidate region.
[0063] (3);
[0064] (4);
[0065] in: This represents the relevance score of the elements in the current candidate region. Indicates the first candidate in the current candidate region Each element interacts with and perceives weight. This indicates the semantic concentration of elements in the current candidate region. Indicates the first candidate in the current candidate region The percentage of the semantic relevance score of each element to the current instruction in the total semantic relevance scores of all elements to the current instruction. This represents the natural exponential function. Indicates temperature parameter, This represents the semantic relevance score between any element in the current set of structured elements and the current instruction.
[0066] Relevance score of current candidate region elements This is used to encourage the search process to prioritize areas where "highly relevant clues mainly correspond to interactive elements," thereby suppressing the interference of decorative elements or static text on area judgment.
[0067] Semantic concentration of elements in the current candidate region This is used to measure whether the correlation distribution within the current candidate region is concentrated or dispersed. When A higher value indicates a clearer semantic focus within the current candidate region, making it more suitable as the input region for subsequent localization; when... A lower correlation indicates a more dispersed distribution of relevance, with a lack of a clear semantic center within the candidate region. If the high correlation in the current candidate region is mainly concentrated on a few key elements, it indicates that the candidate region has a clearer semantic focus and is more suitable as a localization region.
[0068] Specifically, this step can be completed by the dynamic perception action module, which uses the following method to generate new candidate regions by performing one of three types of perception actions—focusing, shifting, or expanding—based on the structured element set and the semantic relevance score of each element to the instruction:
[0069] S311: Compare the number of elements in the current candidate region with the number of elements in the previous round candidate region. If the number of elements in the current candidate region is greater than or equal to 90% of the number of elements in the previous round candidate region, then perform a focusing action, select elements from the current candidate region that have a higher semantic relevance score to the instruction, calculate the minimum enclosing region of the selected elements, and use the minimum enclosing region of the selected elements as a new candidate region. If the number of elements in the current candidate region is less than 90% of the number of elements in the previous round candidate region, then proceed to the next step.
[0070] The focusing action is used to narrow the perception range when the elements in the current candidate region are relatively dense, so that the search is further focused on the local area most relevant to the user's instructions, avoiding the extraction of too large an area, which would not be of significant help to subsequent reasoning.
[0071] S312: Find elements outside the current candidate region that have a high semantic relevance score to the instruction, perform a transition action, calculate the minimum enclosing region of the selected element, and use the minimum enclosing region as a new candidate region, or perform an expansion action to merge the selected element with the elements in the current candidate region to form a new candidate region.
[0072] The shift action can redirect the search from candidate regions with insufficient information to regions more likely to contain the target, preventing the search path from being confined to a single local area. The expand action is applied when the current candidate region is too narrow and lacks sufficient context, restoring a larger range of interface information and preventing the search process from getting stuck in an overly narrow region prematurely, thus providing the necessary contextual information for subsequent planning.
[0073] By introducing three perceptual actions—focusing, shifting, and expanding—the observation range can be dynamically adjusted according to the strength of cues in the current candidate region. This allows for the gradual search and filtering of candidate regions most relevant to the instruction, thereby improving the accuracy and stability of target localization. It avoids the problem of difficulty in recovering from early deviations in the fixed-path gradual cropping method and solves the problem that existing methods are easily affected by irrelevant controls, redundant text, and visual clutter in high-resolution screenshots, making it difficult to stably lock onto the target region.
[0074] S4: Quality scores are given to all candidate regions based on the relevance score of elements in the candidate region or new candidate region, the consistency of interface coverage within the candidate region or new candidate region, and the semantic concentration.
[0075] Specifically, the quality of all candidate regions can be scored according to equation (5):
[0076] (5);
[0077] in: This indicates the quality score of the candidate region. This indicates the relative contribution of the element's correlation score. This represents the relative contribution to the consistency of interface coverage within the current candidate region. This represents the relative contribution of the semantic concentration of the elements in the current candidate region.
[0078] In formula (5) , , A set of optimal values can be obtained through comparative experiments. The regional quality assessment table is shown in Table 1:
[0079] Table 1
[0080]
[0081] This step can be completed through the region quality assessment module. This module comprehensively evaluates candidate regions from three aspects, providing a stable basis for subsequent searches and final region selection. First, it evaluates the relevance between interface elements and user commands within the candidate regions, assigning higher weight to interactive elements to highlight areas more likely to be targets. Second, it evaluates the coverage of the region space by actual interface elements within the candidate regions to avoid selecting areas primarily composed of background, white space, or decorative content. Third, it evaluates the semantic concentration within the candidate regions, i.e., whether highly relevant elements form a relatively compact and clear semantic aggregation within the region, rather than being scattered and mixed in multiple irrelevant locations. Based on the above evaluation results, the region quality assessment module generates a candidate region quality score and sends this score to the action planning module for candidate region selection, comparison, and search path updates.
[0082] S5: Plan and search for candidate regions generated by different sensory actions. Based on the current candidate region status, candidate actions, newly generated candidate regions, and quality scores of all candidate regions, construct a region search tree, select the optimal search path within a given search budget, and output the best region.
[0083] An optimized approach can be to use a Monte Carlo tree search strategy to plan and search for candidate regions generated by different sensory actions.
[0084] The optimized current candidate region state includes the current candidate region, the structured element set of the current candidate region, and the semantic relevance score of each element in the structured element set of the current candidate region to the instruction.
[0085] Specifically, the method for constructing a region search tree, selecting the optimal search path within a given search budget, and outputting the best region is as follows:
[0086] S511: Each node in the search tree represents a candidate region state, and the edges in the search tree represent perceived actions. Starting from the root node, the most promising action branch is selected according to the Monte Carlo tree search strategy.
[0087] S512: After reaching the leaf node, generate new candidate regions for the perception actions that have not yet been expanded, and add them to the search tree;
[0088] S513: Use the quality score of the candidate region as the reward of the leaf node, and backpropagate the reward of the leaf node along the search path to update the statistics of each node in the search tree.
[0089] S514: After multiple rounds of searching, select the candidate region with the highest quality score or the most visits from the search tree as the best region.
[0090] To address the issue that relying solely on local greedy scaling in complex interfaces can easily lead to erroneous paths, this invention models the region selection process as a planning search problem and combines it with Monte Carlo tree search to schedule different perceptual actions. This allows the system to weigh multiple candidate search paths, thereby more effectively finding the high-quality region most relevant to the instruction. Unlike directly predicting the entire screen screenshot at once, this invention models the region search process as a reversible search tree structure, enabling the system to explore, compare, and correct between different region states, thus improving the positioning stability in complex interfaces.
[0091] S6: Input the optimal region and natural language instructions into the basic multimodal large model, and output the location prediction results of the target element.
[0092] After obtaining the optimal region, the basic multimodal large model is no longer required to directly locate the entire high-resolution interface screenshot. Instead, the optimal region and user commands are input into the basic multimodal large model, allowing it to complete the final target localization only within that region. This process corresponds to the "search first, predict later" overall framework of this invention, that is, first determining the most suitable local region for localization through a region search strategy, and then performing coordinate prediction within that region. Addressing the issue that many existing methods rely on additional training or specific model adaptation, this invention adopts a training-independent dynamic region search framework, which can be plugged into existing multimodal large models as a plug-and-play module. This improves their localization performance and generalization ability in real-world interface scenarios without retraining or fine-tuning, thus enhancing the interface localization performance of existing models.
[0093] System experiments were conducted using two evaluation datasets, ScreenSpot and ScreenSpot-Pro, to verify the effectiveness of this invention in real-world interface automatic localization tasks. The ScreenSpot dataset targets general graphical user interface environments, covering mobile application interfaces and some desktop system interfaces, including various real-world usage scenarios such as social applications, e-commerce platforms, system settings pages, content browsing pages, and form filling pages. The interfaces in this dataset exhibit significant differences in style, including text-based function menu pages and icon-based operation panel pages, as well as various interface organization forms such as list structures, card layouts, multi-column layouts, and pop-up interactive interfaces. Each sample consists of a complete screenshot and a corresponding natural language command. The command describes the user's operation target, and the annotation information is given in the form of the bounding box of the target control, thus providing a unified standard for evaluating localization accuracy. The ScreenSpot-Pro dataset is further constructed based on ScreenSpot for highly complex interface environments and is divided into multiple subcategories according to interface type to cover professional scenarios with different structural features and interaction modes. Specifically, these include desktop office software interfaces, graphic design and multimedia editing software interfaces, development and programming tool interfaces, data analysis and visualization platform interfaces, and complex system management backend interfaces. Desktop office software interfaces typically feature multi-level menus, toolbars, and content editing areas, with a large number of elements and clear hierarchical relationships. Graphic design and multimedia editing software interfaces often include densely packed icon-style buttons, parameter adjustment panels, and floating window structures, with small target controls and high visual similarity. Development and programming tool interfaces commonly feature code editing areas, file tree structures, debugging panels, and multiple tabs in parallel layouts, resulting in complex interface space division, with target elements potentially nested within multiple layers of containers. Data analysis and visualization platform interfaces typically include chart areas, filter controls, interactive dashboards, and configuration sidebars, exhibiting strong structural semantic expression and spatial layout. System management backend interfaces primarily use combinations of tables, lists, filter conditions, and operation buttons, with many similar functions potentially existing on the same page, easily leading to semantic confusion. The aforementioned subcategories cover real-world application interfaces under different resolution conditions, control densities, and interaction paradigms, enabling ScreenSpot-Pro to systematically test the model's localization capabilities in multi-level nested structures, densely distributed small-sized controls, and environments with highly similar icons. All subcategory samples employ a unified data organization format: pairing natural language commands with complete screen screenshots and performing precise spatial annotations on the target controls, thus ensuring consistency and comparability in cross-category evaluations.By evaluating the methods under the above subcategories, we can analyze their adaptability in different professional interface scenarios in a more granular way, providing data support for verifying the universality of the present invention in highly complex real interface environments.
[0094] Table 2 presents the comparison results of the present invention and existing models on ScreenSpot-Pro. In terms of overall average accuracy, most models without dynamic region search mechanisms have an average accuracy ranging from 7.7 to 26.8, while DRS-GUI-2B reaches 38.3 and DRS-GUI-7B reaches 45.7, placing them at the forefront. Taking the 7B scale model as an example, DRS-GUI-7B improves accuracy by 18.9 percentage points compared to UGround-V1-7B's 26.8, and by more than 10 percentage points compared to UI-TARS-7B's 35.7, indicating that introducing dynamic region search and planning mechanisms can significantly improve localization stability in high-resolution, element-dense, and complex interfaces. Looking at the results for each subcategory, this improvement maintains a consistent trend across multiple categories, including Development, Creative, Scientific, and OS. For example, in the Scientific category, DRS-GUI-7B achieved an accuracy of 52.4, significantly higher than most of the comparison methods; in the OS category, it achieved 40.3, an improvement of over 20 percentage points compared to some baseline models. This indicates that the method of this invention has strong adaptability in different professional interface scenarios. Table 3 shows the comparison results on ScreenSpot. The average accuracy of DRS-GUI-7B is 89.9, the highest among all comparison methods. It is 3.6 percentage points higher than UGround-V1-7B's 86.3, and 5 percentage points higher than Qwen2.5-VL-7B's 84.9. It also maintains a leading position in desktop text and web icon scenarios, demonstrating stable gains under common interface distributions.
[0095] The results from both datasets show that the improvement of this invention is more significant in highly complex professional interfaces, and it also maintains leading performance in general interface environments, verifying the effectiveness and generalization ability of the "candidate region quality assessment + planning search" mechanism under different interface complexity conditions.
[0096] The results from both datasets show that the improvement of this invention is more significant in highly complex professional interfaces, and it also maintains leading performance in general interface environments, verifying the effectiveness and generalization ability of the "candidate region quality assessment + planning search" mechanism under different interface complexity conditions.
[0097] Table 2
[0098]
[0099]
[0100] Table 3
[0101]
[0102] A graphical user interface instruction localization system based on dynamic region search is used to execute a graphical user interface instruction localization method based on dynamic region search as described in any of the above, comprising a task input module, an interface acquisition module, an element perception module, a dynamic perception action module, a region quality assessment module, an action planning module, and a final localization module.
[0103] The task input module is used to receive natural language instructions input by the user;
[0104] The interface acquisition module is used to acquire a screenshot of the current complete interface.
[0105] The element perception module is used to parse the elements in the candidate area of the current complete interface screenshot, output a structured element set, and calculate the semantic relevance score between the elements and the instructions, the interface coverage consistency within the candidate area, the element relevance score, and the semantic concentration.
[0106] The dynamic perception action module is used to generate new candidate regions by performing one of three types of perception actions—focusing, shifting, or expanding—based on the structured element set and the semantic relevance score between each element and the instruction.
[0107] The regional quality assessment module scores all candidate regions based on element correlation scores, interface coverage consistency within candidate regions, and semantic concentration.
[0108] The action planning module is used to plan and search for candidate regions generated by different perceptual actions. Based on the current candidate region status, candidate actions, newly generated candidate regions, and quality scores of all candidate regions, it constructs a region search tree, selects the optimal search path within a given search budget, and outputs the best region.
[0109] The final localization module is used to input the optimal region and natural language instructions into the basic multimodal large model and output the location prediction results of the target element.
[0110] In summary, this invention constructs a unified element representation by performing structured analysis of interface elements and progressively narrows the search range within a multi-granularity candidate region space, achieving step-by-step localization from global to local. During candidate region selection, a region evaluation mechanism is established, scoring all candidate regions based on element relevance scores, interface coverage consistency within the candidate region, and semantic concentration, thus forming a stable and reliable region scoring basis. Regarding the search strategy, the region selection process is modeled as a tree search problem, using a Monte Carlo tree search mechanism to make decisions on different perceptual actions. This allows the system to utilize high-value region information while maintaining the ability to explore potential regions, effectively balancing search efficiency and localization accuracy. Through the coordinated operation of three perceptual actions, it avoids getting trapped in local optima while maintaining the hierarchical convergence characteristics of the search process. In the final localization stage, the optimal region and natural language instructions are input into a basic multimodal large model, outputting the predicted location of the target element. This method achieves automatic matching and action execution between natural language commands and graphical interface elements without the need for manual rule intervention. It has the advantages of clear structure, strong scalability, and adaptability to complex interface environments, and is suitable for mobile terminals, desktop systems and other device scenarios with graphical user interfaces.
[0111] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A graphical user interface command positioning method based on dynamic region search, characterized in that: Includes the following steps: S1: Receive natural language commands input by the user and obtain a screenshot of the current complete interface; S2: Parse the elements of the candidate area in the current complete interface screenshot, output a structured element set containing bounding boxes, semantic descriptions and interactive attributes, and combine the natural language instructions input by the user. Calculate the semantic relevance score between the element and the instruction based on the semantic description, calculate the interface coverage consistency within the candidate area based on the bounding box, and calculate the element relevance score and semantic concentration within the candidate area based on the semantic relevance score and interactive attributes. S3: Based on the structured element set and the semantic relevance scores of each element and the instruction, perform one of the three types of perception actions: focusing, shifting, or expanding, to generate a new candidate region, and calculate the semantic relevance scores of each element and the instruction in the new candidate region, the interface coverage consistency, the element relevance score, and the semantic concentration in the new candidate region. The following method is used to generate new candidate regions by performing one of three perceptual actions—focusing, shifting, or expanding—based on the structured element set and the semantic relevance score of each element to the instruction: S311: Compare the number of elements in the current candidate region with the number of elements in the previous round candidate region. If the number of elements in the current candidate region is greater than or equal to 90% of the number of elements in the previous round candidate region, then perform a focusing action, select elements from the current candidate region that have a higher semantic relevance score to the instruction, calculate the minimum enclosing region of the selected elements, and use the minimum enclosing region of the selected elements as a new candidate region. If the number of elements in the current candidate region is less than 90% of the number of elements in the previous round candidate region, then proceed to the next step. S312: Find elements outside the current candidate region that have a high semantic relevance score to the instruction, perform a transition action, calculate the minimum enclosing region of the selected element, and use the minimum enclosing region as a new candidate region, or perform an expansion action to merge the selected element with the elements in the current candidate region to form a new candidate region. S4: Quality scores are given to all candidate regions based on the relevance score of elements in the candidate region or new candidate region, the consistency of interface coverage within the candidate region or new candidate region, and the semantic concentration. S5: Plan and search for candidate regions generated by different sensory actions. Based on the current candidate region status, candidate actions, newly generated candidate regions, and quality scores of all candidate regions, construct a region search tree, select the optimal search path within a given search budget, and output the best region. S6: Input the optimal region and natural language instructions into the basic multimodal large model, and output the location prediction results of the target element.
2. The graphical user interface command positioning method based on dynamic region search according to claim 1, characterized in that: In steps S2 and S3, the semantic relevance score between each element and the instruction is calculated according to equation (1): （1）； in: Indicates the first The semantic relevance score of each element to the current instruction. Indicates user commands, Indicates matrix transpose. Indicates a text encoder. Indicates the first Each element constructs the corresponding text description. This represents the L2 norm.
3. The graphical user interface command positioning method based on dynamic region search according to claim 2, characterized in that: In steps S2 and S3, the interface coverage consistency within the current candidate region is calculated according to equation (2): （2）； in: This indicates that the interface coverage within the current candidate region is consistent. Indicates the current candidate region. Indicates area, Indicates the first candidate in the current candidate region The bounding box coordinates of each element. This indicates the total number of elements in the current candidate region.
4. The graphical user interface command positioning method based on dynamic region search according to claim 3, characterized in that: In steps S2 and S3, the element relevance score is calculated according to equation (3), and the semantic concentration is calculated according to equation (4): （3）； （4）； in: This represents the relevance score of the elements in the current candidate region. Indicates the first candidate in the current candidate region Each element interacts with and perceives weight. This represents a constant that avoids a denominator of zero. This indicates the semantic concentration of elements in the current candidate region. Indicates the first candidate in the current candidate region The percentage of the semantic relevance score of each element to the current instruction in the total semantic relevance scores of all elements to the current instruction. This represents the natural exponential function. Indicates temperature parameter, This represents the semantic relevance score between any element in the current set of structured elements and the current instruction.
5. The graphical user interface command positioning method based on dynamic region search according to claim 4, characterized in that: In step S4, quality scores are calculated for all candidate regions according to equation (5): （5）； in: This indicates the quality score of the candidate region. This indicates the relative contribution of the element's correlation score. This represents the relative contribution to the consistency of interface coverage within the current candidate region. This represents the relative contribution of the semantic concentration of the elements in the current candidate region.
6. The graphical user interface command positioning method based on dynamic region search according to claim 1, characterized in that: In step S5, a Monte Carlo tree search strategy is used to plan and search for candidate regions generated by different perceptual actions.
7. The graphical user interface command positioning method based on dynamic region search according to claim 1, characterized in that: The current candidate region state mentioned in step S5 includes the current candidate region, the structured element set of the current candidate region, and the semantic relevance score of each element in the structured element set of the current candidate region to the instruction.
8. The graphical user interface command positioning method based on dynamic region search according to claim 7, characterized in that: The method for constructing the region search tree in step S5, selecting the optimal search path within a given search budget, and outputting the best region is as follows: S511: Each node in the search tree represents a candidate region state, and the edges in the search tree represent perceived actions. Starting from the root node, the most promising action branch is selected according to the Monte Carlo tree search strategy. S512: After reaching the leaf node, generate new candidate regions for the perception actions that have not yet been expanded, and add them to the search tree; S513: Use the quality score of the candidate region as the reward of the leaf node, and backpropagate the reward of the leaf node along the search path to update the statistics of each node in the search tree. S514: After multiple rounds of searching, select the candidate region with the highest quality score or the most visits from the search tree as the best region.
9. A graphical user interface command positioning system based on dynamic region search, used to execute a graphical user interface command positioning method based on dynamic region search as described in any one of claims 1 to 8, characterized in that, It includes a task input module, an interface acquisition module, an element perception module, a dynamic perception action module, an area quality assessment module, an action planning module, and a final positioning module; The task input module is used to receive natural language instructions input by the user; The interface acquisition module is used to acquire a screenshot of the current complete interface. The element perception module is used to parse the elements in the candidate area of the current complete interface screenshot, output a structured element set, and calculate the semantic relevance score between the elements and the instructions, the interface coverage consistency within the candidate area, the element relevance score, and the semantic concentration. The dynamic perception action module is used to generate new candidate regions by performing one of three types of perception actions—focusing, shifting, or expanding—based on the structured element set and the semantic relevance score between each element and the instruction. The regional quality assessment module scores all candidate regions based on element correlation scores, interface coverage consistency within candidate regions, and semantic concentration. The action planning module is used to plan and search for candidate regions generated by different perceptual actions. Based on the current candidate region status, candidate actions, newly generated candidate regions, and quality scores of all candidate regions, it constructs a region search tree, selects the optimal search path within a given search budget, and outputs the best region. The final localization module is used to input the optimal region and natural language instructions into the basic multimodal large model and output the location prediction results of the target element.