A method for improving GUI positioning ability without training based on operation chain

Abstract

This invention discloses a training-free method for improving GUI localization capabilities based on an operation chain, relating to the fields of human-computer interaction and intelligent software agent technology. The method includes: acquiring a target interface image and natural language commands; obtaining initial candidate regions through a basic localization model; performing local cropping followed by secondary prediction and mapping back to the original image to update the region; applying a dynamic mask to the image and repeatedly predicting to generate spatially mutually exclusive candidate boxes; constructing a labeled comparison image; combining the input error correction model with commands and rule-based prompts; and selecting the optimal candidate box as the final localization result. This invention significantly reduces accuracy and ambiguity biases in localization by performing a series of interpretable inference time-series operations on the input screenshot and natural language commands, thereby improving the executable localization capabilities and stability of desktop intelligent agents on edge devices such as computers and mobile phones.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction and intelligent software agent technology, and in particular to a method for improving GUI localization capabilities without training based on operation chains. Background Technology

[0002] Graphical user interface intelligent agents, as a core technology in the field of human-computer interaction, are widely used in desktop office, mobile terminals, and professional software automation scenarios. With the development of multimodal large model technology, existing positioning systems, through the collaborative operation of visual perception, semantic understanding, and coordinate regression, construct a complete process from instruction parsing to control mapping. Specifically, this technical solution covers key aspects such as high-resolution screenshot acquisition, dense element feature extraction, and absolute coordinate prediction, aiming to achieve accurate operation under complex workflows.

[0003] However, existing localization methods directly adopt the whole-image regression strategy without specifically suppressing long-distance quantization errors and contextual ambiguity. This may lead to large dispersion of small target coordinates in high-resolution scenes, or selection bias when similar controls coexist, which seriously affects the execution accuracy and robustness of intelligent agents in professional software such as IDEs and EDA. Summary of the Invention

[0004] The main objective of this invention is to provide a method for improving GUI localization capabilities without training, based on an operation chain.

[0005] Another objective of this invention is to provide a device for improving GUI localization capabilities without training based on an operation chain.

[0006] The third objective of this invention is to provide an electronic device.

[0007] The fourth objective of this invention is to provide a non-transitory computer-readable storage medium.

[0008] To achieve the above objectives, a first aspect of the present invention proposes a method for improving GUI localization capabilities without training based on operation chains, comprising:

[0009] S1, acquire the original image and natural language command of the target interface, input the original image and natural language command into the basic localization model for global prediction, and parse the output result to obtain the initial candidate region; S2, perform local cropping operation on the original image based on the initial candidate region to generate a local sub-image, perform prediction again in the coordinate system of the local sub-image to obtain the fine localization result, and inversely map the fine localization result back to the global coordinate system of the original image to update the candidate region; S3 applies a dynamic mask to the input image of the current round, occluding the generated historical candidate regions. The masked image and natural language instructions are then input into the basic localization model for repeated prediction, generating multiple sets of candidate boxes that are mutually exclusive in spatial distribution. S4. Construct a comparison image containing visual annotation information of each candidate box in the candidate box set. Input the comparison image, natural language instructions and preset selection rule prompts into the error correction selection model. Through comparison and evaluation, determine the optimal candidate box from the candidate box set as the final localization result.

[0010] Optionally, the original image and natural language command of the target interface are obtained, and the original image and natural language command are input into the basic localization model for global prediction. The output results are parsed to obtain the initial candidate region, including: The original image is used as the initial input image and the current iteration step is set to 1. The initial input image and the natural language instruction are input into the basic localization model that supports bounding box coordinate output or click point coordinate output. The text sequence containing coordinate position information is generated by autoregressive decoding. The text sequence is parsed using a predefined regular expression extraction function. If the basic localization model is a bounding box output type, the coordinates of the top left and bottom right corners are extracted and the center coordinates are calculated to construct an initial candidate bounding box. If the basic localization model is a click point output type, the coordinates of the target center point are directly extracted and the initial candidate bounding box with spatial scale is generated by expanding outward from the center of the target point with a preset pixel threshold. The generated initial candidate bounding boxes are used as the winning prediction boxes for the current round to trigger subsequent local cropping operations to eliminate long-distance quantization errors caused by direct regression of coordinates from the large-resolution whole image.

[0011] Optionally, a local cropping operation is performed on the original image based on the initial candidate regions to generate local sub-images. Prediction is then performed again in the coordinate system of these local sub-images to obtain fine-grained localization results. These fine-grained localization results are then inversely mapped back to the global coordinate system of the original image to update the candidate regions, including: Obtain the winning prediction box generated in the previous step and use the center point of the box as the cropping origin, according to the preset cropping ratio. A sub-image is extracted from the current input image, and the sub-image is enlarged and re-input into the basic localization model to make predictions in a local relative coordinate system with higher resolution and smaller quantization error; After obtaining the fine coordinate prediction results in the local relative coordinate system, based on the absolute offset recorded during the clipping operation ( , ) and scaling factor Perform an inverse affine transformation to map the fine coordinate prediction results back to the global coordinate system of the original full image to obtain the actual executable coordinates; Determine whether the current iteration step has reached the preset number of coarse-to-fine iteration focusing times N. If not, select the winning box of the current round as the new anchor point and return to perform the local clipping operation. If it has reached the target, extract the center point coordinates of the final winning box and map them back to the original image dimension as the final execution anchor point output of the graphical user interface agent.

[0012] Optionally, a dynamic mask is applied to the input image of the current round to occlude the generated historical candidate regions. The masked image is then input into the basic localization model along with the natural language instructions for repeated prediction, generating multiple spatially exclusive candidate box sets, including: Initialize the candidate set for the current round to an empty set, enter the candidate generation inner loop and set the loop variable. For each loop, call the mask function to reset the pixel values ​​of all historical candidate box regions that already exist in the candidate set in the current input image to zero vectors to generate pure black occlusion areas, thereby obtaining the mask image. The natural language instructions and the mask image are re-input into the basic localization model for inference calculation to obtain new candidate boxes and add them to the candidate set, ensuring that the newly generated candidate boxes do not overlap with existing erroneous candidates; Repeat the mask image construction and re-prediction accumulation steps until the loop ends to obtain a specified number of candidate boxes that are mutually exclusive in spatial distribution and semantically discriminative, in order to overcome the ambiguity bias caused by the multimodal model ignoring the actual physical space Euclidean distance.

[0013] Optionally, a comparison image containing visual annotation information of each candidate box in the candidate box set is constructed. The comparison image, natural language instructions, and preset selection rule prompts are input into the error correction selection model. The optimal candidate box is determined from the candidate box set as the final localization result through comparative evaluation, including: On the original image copy, each candidate box in the candidate box set is explicitly rendered using different geometric color markers and numerical numbers, generating a multi-candidate comparison image with labeled visual cues to provide an intuitive spatial comparison basis; Construct structured prompts that include mandatory interface selection principles and thought chains. The selection principles stipulate that functional attribute matching takes priority over other attributes, interactive controls take priority over static text components, and define the interface element quality hierarchy as follows: icon combined with text area is better than pure complete icon, pure complete icon is better than pure complete text, and pure complete text is better than incomplete area of ​​multiple elements stacked. The natural language instruction, the multi-candidate comparison image, and the structured prompt words are input into an error-correcting selection model with visual question-answering capabilities. The error-correcting selection model calculates the posterior preference probability of each candidate box based on the evaluation weight of the prompt words, and outputs the winning candidate box identifier that maximizes the probability as the final localization result after deambiguity biasing.

[0014] To achieve the above objectives, a second aspect of the present invention provides an apparatus for improving GUI localization capabilities without training based on an operation chain, comprising: The prediction module is used to acquire the original image and natural language command of the target interface, input the original image and natural language command into the basic localization model for global prediction, and parse the output results to obtain the initial candidate region. The mapping module is used to perform local cropping operations on the original image based on the initial candidate region to generate a local sub-image, perform a second prediction in the coordinate system of the local sub-image to obtain a fine localization result, and then backmap the fine localization result back to the global coordinate system of the original image to update the candidate region. The masking module is used to apply dynamic masking to the input image of the current round, occluding the generated historical candidate regions. The masked image and natural language instructions are then input into the basic localization model for repeated prediction, generating multiple sets of candidate boxes that are mutually exclusive in spatial distribution. The evaluation module is used to construct a comparison image containing visual annotation information of each candidate box in the candidate box set. The comparison image, natural language instructions, and preset selection rule prompts are input into the error correction selection model. The optimal candidate box is determined from the candidate box set as the final localization result through comparison and evaluation.

[0015] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0016] To achieve the above objectives, a third aspect of this application provides an electronic device, including a processor and a memory; wherein the processor reads executable program code stored in the memory to run a program corresponding to the executable program code, for implementing the method for improving GUI positioning capabilities without training based on the operation chain as described in the first aspect embodiment.

[0017] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements the method for improving GUI localization capabilities without training based on an operation chain as described in the first aspect embodiment.

[0018] The embodiments of the present invention have the following beneficial effects: This invention effectively eliminates coordinate quantization errors and interference from similar controls through coarse-to-fine focusing iteration and mask re-prediction mechanisms, significantly improving the accuracy and robustness of element localization in high-resolution complex interfaces. This method suppresses accuracy and ambiguity bias without additional training, achieving plug-and-play enhancement of cross-platform GUI agent localization capabilities. Attached Figure Description

[0019] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart illustrating a method for improving GUI localization capabilities without training based on an operation chain, provided as an embodiment of the present invention; Figure 2 A flowchart of another method for improving GUI localization capabilities without training based on an operation chain, provided in an embodiment of the present invention; Figure 3 This is an example of an iterative focusing map from coarse to fine provided in an embodiment of the present invention. Figure 4 The mask and re-prediction branch diagram provided in the embodiments of the present invention; Figure 5 This is a visualization of the GUI executable effect provided in the embodiments of the present invention; Figure 6 This is a structural diagram of a device for improving GUI localization capabilities without training based on an operation chain, provided in an embodiment of the present invention. Detailed Implementation

[0020] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0021] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0022] The following describes, with reference to the accompanying drawings, a method and apparatus for improving GUI localization capabilities without training based on an operation chain, according to embodiments of the present invention.

[0023] Example 1 This invention provides a method for improving GUI localization capabilities without training based on operation chains, such as... Figure 1 As shown, the method includes the following steps: S1: Obtain the original image and natural language command of the target interface, input the original image and natural language command into the basic localization model for global prediction, and parse the output result to obtain the initial candidate region.

[0024] In order to complete the initial data collection and coarse localization for GUI localization and lay the foundation for subsequent processes, this application generates initial candidate bounding boxes by setting initialization specifications, inputting relevant data, and performing model prediction and analysis.

[0025] In this embodiment, the primary task is to complete the collection of basic data and initial positioning, laying a solid foundation for subsequent refined positioning and ambiguity resolution, and ensuring the orderly progress of the entire positioning process. In this embodiment, initialization and input specifications are prerequisites for successful execution. This application strictly adheres to preset standards in setting various parameters and models to avoid positioning deviations or target loss due to unreasonable parameters. This application first reads the original full-screen screenshot image I of the target smart device (such as a computer, mobile phone, tablet, etc.), where the dimensions of image I are clearly defined. Here, H represents the image height, W represents the image width, and 3 represents the RGB three channels, ensuring the integrity and accuracy of image information; at the same time, it accurately obtains the natural language command q input by the user, which clearly defines the target requirements of GUI positioning, such as "click the confirmation button in the upper right corner of the interface" or "the boundary range of the positioning settings menu", providing clear directional guidance for subsequent model prediction.

[0026] In this application embodiment, a pre-trained basic multimodal GUI localization model M is carefully selected as the basic localization model. This model can flexibly choose between bounding box output type or coordinate point output type, without the need for additional fine-tuning training for specific GUI interfaces. It can adapt to various common GUI scenarios, significantly reducing operational complexity. Simultaneously, this application rigorously initializes a series of algorithm hyperparameters. The number of coarse-to-fine iteration focusing times N is set to 2 by default, ensuring localization accuracy while avoiding target loss due to excessive iterations. The image local cropping ratio λ is set between 0.5 and 0.7 to ensure that the cropped sub-image retains sufficient contextual information while avoiding target loss due to over-cropping. Parameters such as the number of mutually exclusive candidate boxes generated in each round, mask intensity m, and candidate voting rounds T have all been verified through extensive experiments to balance localization accuracy and computational efficiency, avoiding localization deviations caused by unreasonable parameters. Furthermore, this application specifically defines an error-correcting selection model m for candidate result comparison and evaluation. This model possesses powerful visual question-answering capabilities, providing core support for subsequent "selection of the best" and disambiguation.

[0027] In this embodiment of the application, the specific execution process of S1 strictly follows the preset logic, firstly processing the original image... With natural language instructions Input the basic localization model together This model generates text sequences containing coordinate information through autoregressive decoding. This sequence carries the positional information of the target GUI element and cannot be directly used for location execution. Therefore, this application specifically defines a regular expression extraction function. This is used to accurately extract location information from a text sequence. In this embodiment, based on the type of the basic location model, there are two extraction logics: if the basic location model is a bounding box output type, the regular expression extraction function will parse out the top left corner of the target bounding box (…). , ) and the bottom right corner ( , The coordinates of the bounding box center are calculated using the following formula:

[0028] The initial candidate bounding boxes are finally formed. If the basic positioning model is a click-point output type, after extracting the coordinates of the target center point, use... =25 pixels is the diffusion threshold, calculated using the formula Generate candidate bounding boxes with spatial scale to ensure that subsequent positioning has a clear spatial range to support it.

[0029] In this embodiment of the application, by initializing the input specifications and carrying out basic data collection and model prediction analysis, the foundation is laid for subsequent predictions.

[0030] S2: Based on the initial candidate region, perform a local cropping operation on the original image to generate a local sub-image. Perform a second prediction in the coordinate system of the local sub-image to obtain a fine localization result. Then, reverse map the fine localization result back to the global coordinate system of the original image to update the candidate region.

[0031] To eliminate long-distance quantization errors caused by global positioning, this application achieves precise GUI positioning from coarse to fine through iterative local clipping and inverse coordinate mapping.

[0032] In this embodiment, S2 is the core component for achieving "coarse-to-fine" localization. Its core idea is to transform global localization into precise local localization through local cropping and coordinate mapping, thus completely solving the accuracy problem caused by insufficient global prediction in large images. In this embodiment, the specific execution process of S2 strictly follows preset logic. First, it obtains the previously generated initial candidate bounding boxes, which serve as anchor points for local cropping, clarifying the cropping range and benchmark. The cropping ratio λ set in this application ranges from 0.5 to 0.7, preferably λ=0.6. This ratio has been verified through extensive experiments, ensuring that the local sub-image contains sufficient contextual information while effectively reducing the localization range and improving localization accuracy, avoiding information redundancy due to an excessively large cropping range or target loss due to an excessively small cropping range.

[0033] In this embodiment, the generation of the local sub-image strictly follows the principle of "centered on the initial candidate region." That is, around the center point of the generated initial candidate box, a certain range of the image surrounding that center point is captured as the local sub-image. This ensures that the sub-image contains complete target elements and necessary contextual information, providing support for subsequent fine-grained localization. Furthermore, based on the local sub-image, this application re-invokes the basic localization model for refined prediction, optimizes the localization results through autoregressive decoding, corrects potential biases in the initial prediction, and further improves the accuracy of localization.

[0034] Inverse coordinate mapping is a core technical point. This application achieves a precise correspondence between local sub-image coordinates and global coordinates through strict coordinate transformation logic. Specifically, the fine-grained positioning coordinates obtained in the local sub-image need to be converted into the global coordinates of the original image through inverse mapping. The core formula follows: Global coordinates = Crop origin coordinates + Local sub-image coordinates × Crop ratio λ. This formula ensures that the positioning results in the local sub-image can accurately correspond to the global coordinate system of the original image, avoiding coordinate deviations.

[0035] In this embodiment of the application, the criteria for determining iterative convergence are clear: when the current iteration step reaches a preset value of N (default N=2), and the Euclidean distance of the localization result tends to stabilize and no longer significantly improves, it is determined to be convergent, and the iteration stops. Figure 3 The diagram shows the iterative focusing pattern from coarse to fine, clearly illustrating the iterative process of S2. From the initial global coarse positioning to the fine positioning after the first local cropping, and finally to the precise positioning, each step improves the positioning accuracy, fully demonstrating the "coarse to fine" positioning logic of this application.

[0036] In this embodiment, fine-grained GUI positioning is achieved through iterative local cropping, inverse coordinate mapping, and convergence determination, laying the foundation for the subsequent generation of multiple mutually exclusive candidate box sets in spatial distribution.

[0037] S3 applies a dynamic mask to the input image of the current round, occluding the generated historical candidate regions. The masked image and the natural language command are then input into the basic localization model for repeated prediction, generating multiple sets of candidate boxes that are mutually exclusive in spatial distribution.

[0038] To overcome the ambiguity bias of multimodal models and reduce localization interference, this application generates a set of spatially mutually exclusive and semantically distinguishable candidate boxes through dynamic masking and repeated prediction.

[0039] In this application embodiment, the core of step S3 is to generate a set of candidate boxes that meet the requirements through dynamic masking technology and repeated prediction, providing sufficient samples for subsequent comparison and selection. This application first clarifies that the core objective of step S3 is to overcome the semantic ambiguity that may exist in multimodal models. Through masking and repeated prediction, spatially mutually exclusive and semantically discriminative candidate boxes are generated, avoiding localization bias caused by similar elements, while providing rich candidate samples for subsequent "selection of the best among the best".

[0040] In this application embodiment, to ensure the uniqueness and distinguishability of candidate boxes, this application uses dynamic masking technology to perform targeted processing on the input image. The constructed mask can accurately identify historical candidate box regions, set their pixel values ​​to zero, and form pure black occlusion areas, thereby preventing newly generated candidate boxes from overlapping with existing candidate boxes and achieving spatial mutual exclusion. Specifically, the masking function defined in this application can automatically identify historical candidate box regions and perform occlusion processing on these regions to ensure that newly generated candidate boxes do not overlap with existing candidate boxes, thus generating a set of candidate boxes that are spatially independent and semantically distinct.

[0041] In this embodiment, the mask is constructed strictly according to the principle of practicality, and its range and intensity are dynamically adjusted according to the candidate set in the current round. When making the first prediction, since the candidate set is empty, the mask does not play a role, and the input image is the original image. After multiple rounds of prediction, the mask will automatically occlude the generated candidate box areas to ensure that the newly generated candidate boxes can exist independently and avoid stacking and overlapping, thereby ensuring the diversity and effectiveness of the candidate set.

[0042] Meanwhile, this application combines the fine positioning results of S2 to further optimize the mask construction logic. Based on the existing accurate positioning information, it performs targeted mask processing on the input image, grayscales, blurs, or covers irrelevant areas, retains the area information related to the target, reduces irrelevant interference, and ensures that the generated candidate boxes can accurately match the user's needs, providing high-quality candidate samples for subsequent comparison and selection.

[0043] In this embodiment, a mask image is constructed by dynamically masking historical candidate regions and combining it with fine localization results. A set of spatially mutually exclusive and semantically discriminative candidate boxes is generated cyclically, laying the foundation for determining the optimal candidate box in the future.

[0044] S4. Construct a comparison image containing visual annotation information of each candidate box in the candidate box set. Input the comparison image, natural language instructions and preset selection rule prompts into the error correction selection model. Through comparison and evaluation, determine the optimal candidate box from the candidate box set as the final localization result.

[0045] To eliminate the ambiguity bias of multimodal models and improve positioning accuracy, this application constructs comparison images, sets constraint rules, and optimizes the selection mechanism to ensure that the GUI positioning results are accurate and reliable, meeting the needs of actual operation.

[0046] In this embodiment, the specific execution process of S4 is divided into three core steps. The first step is to construct the comparison image. In this embodiment, the application uses different geometric colors and numerical numbers to explicitly render all generated candidate boxes on a copy of the original image. Each candidate box is clearly labeled, including the coordinate range and corresponding function of the candidate box. Through clear visual distinction, it provides an intuitive basis for the comparative evaluation of the error correction selection model, avoids selection errors caused by the close position and semantic similarity of candidate boxes, and ensures that the error correction selection model can clearly distinguish the differences between different candidate boxes.

[0047] In this embodiment, the second step is structured prompt word constraint. In order to correct the inherent selection tendency of the visual language model and avoid its over-reliance on text semantic similarity and neglect of the interaction logic of the GUI interface, this application injects a set of mandatory interface selection rules as prompt words through text, providing a clear standard for the evaluation of the error correction selection model.

[0048] In this embodiment, the prompt words include three core constraint rules: first, functional attribute matching priority, that is, the function of the GUI element corresponding to the candidate box must be highly matched with the user's input natural language command requirements; second, interaction priority constraint, that is, interactive controls take precedence over non-interactive elements such as static text and images; and third, element quality level constraint, that is, "icon + text" combination elements are superior to pure icon elements, and pure icon elements are superior to pure text elements, ensuring that the evaluation logic of the error correction selection model meets the actual needs of GUI operation.

[0049] In this embodiment, the third step is error correction selection model inference. Natural language instructions, labeled comparison images, and structured prompts are input into the error correction selection model. Based on preset constraint rules, the model comprehensively evaluates all candidate boxes, calculates the matching probability of each candidate box, and obtains the candidate box with the highest posterior preference probability through a formula, which is then selected as the optimal candidate box.

[0050] in, This represents the optimal decision in the t-th iteration. Let be the set of candidate solutions for round t. To meet the target requirements, The dataset labeled in round t. This is the optimal parameter set.

[0051] In this embodiment, if the current iteration round has not reached the preset number of iterations, the application uses the optimal candidate box as the anchor point and returns to S2 for further refined iteration optimization; if the preset number of iterations has been reached, the application extracts the center point coordinates of the optimal candidate box, transforms them to the original global coordinate system through coordinate inverse mapping, and uses them as the final execution anchor point of the GUI agent to complete the positioning process.

[0052] In this embodiment, step S4 further includes the visualization of the localization results. This application achieves an intuitive display of the localization results by marking the final localization points and candidate box comparison markers on the original image, while combining... Figure 5 The GUI executable visualization shown clearly presents the final location and coordinate information of the GUI elements, ensuring that the location results are traceable and verifiable.

[0053] In this embodiment, by constructing a comparison image, setting selection rules, and combining an error correction model, the candidate box is accurately screened, ambiguity is completely eliminated, and the GUI positioning results are ensured to be accurate and reliable. At the same time, the positioning accuracy is improved through visualization.

[0054] Example 2 This invention provides an apparatus 10 for improving GUI localization capabilities without training based on operation chains, such as... Figure 6 As shown, the device includes: The prediction module 100 is used to acquire the original image and natural language command of the target interface, input the original image and natural language command into the basic localization model for global prediction, and parse the output result to obtain the initial candidate region. The mapping module 200 is used to perform a local cropping operation on the original image based on the initial candidate region to generate a local sub-image, perform a re-prediction in the coordinate system of the local sub-image to obtain a fine localization result, and inversely map the fine localization result back to the global coordinate system of the original image to update the candidate region. The mask processing module 300 is used to apply dynamic mask processing to the input image of the current round, occluding the generated historical candidate regions, and repeatedly predicting the masked image and natural language instructions into the basic localization model to generate multiple sets of candidate boxes that are mutually exclusive in spatial distribution. The evaluation module 400 is used to construct a comparison image containing visual annotation information of each candidate box in the candidate box set. The comparison image, natural language instructions and preset selection rule prompts are input into the error correction selection model. The optimal candidate box is determined from the candidate box set as the final localization result through comparison and evaluation.

[0055] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0056] Example 3 To implement the methods of the above embodiments, the present invention also provides an electronic device, which includes a memory and a processor; wherein the processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the various steps of the methods described above.

[0057] Example 4 To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in the foregoing embodiments.

[0058] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0059] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0060] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A method for improving GUI localization capabilities without training based on operation chains, characterized in that, include: S1, acquire the original image and natural language command of the target interface, input the original image and natural language command into the basic localization model for global prediction, and parse the output result to obtain the initial candidate region; S2, perform local cropping operation on the original image based on the initial candidate region to generate a local sub-image, perform prediction again in the coordinate system of the local sub-image to obtain the fine localization result, and inversely map the fine localization result back to the global coordinate system of the original image to update the candidate region; S3 applies a dynamic mask to the input image of the current round, occluding the generated historical candidate regions. The masked image and natural language instructions are then input into the basic localization model for repeated prediction, generating multiple sets of candidate boxes that are mutually exclusive in spatial distribution. S4. Construct a comparison image containing visual annotation information of each candidate box in the candidate box set. Input the comparison image, natural language instructions and preset selection rule prompts into the error correction selection model. Through comparison and evaluation, determine the optimal candidate box from the candidate box set as the final localization result.

2. The method according to claim 1, characterized in that, The process involves acquiring the original image and natural language commands of the target interface, inputting the original image and natural language commands into the basic localization model for global prediction, and parsing the output results to obtain initial candidate regions, including: The original image is used as the initial input image and the current iteration step is set to 1. The initial input image and the natural language instruction are input into the basic localization model that supports bounding box coordinate output or click point coordinate output. The text sequence containing coordinate position information is generated by autoregressive decoding. The text sequence is parsed using a predefined regular expression extraction function. If the basic localization model is a bounding box output type, the coordinates of the top left and bottom right corners are extracted and the center coordinates are calculated to construct an initial candidate bounding box. If the basic localization model is a click point output type, the coordinates of the target center point are directly extracted and the initial candidate bounding box with spatial scale is generated by expanding outward from the center of the target point with a preset pixel threshold. The generated initial candidate bounding boxes are used as the winning prediction boxes for the current round to trigger subsequent local cropping operations to eliminate long-distance quantization errors caused by direct regression of coordinates from the large-resolution whole image.

3. The method according to claim 1, characterized in that, The process of performing a local cropping operation on the original image based on the initial candidate region to generate a local sub-image, performing a second prediction in the coordinate system of the local sub-image to obtain a fine localization result, and then mapping the fine localization result back to the global coordinate system of the original image to update the candidate region includes: Obtain the winning prediction box generated in the previous step and use the center point of the box as the cropping origin, according to the preset cropping ratio. A sub-image is extracted from the current input image, and the sub-image is enlarged and re-input into the basic localization model to make predictions in a local relative coordinate system with higher resolution and smaller quantization error; After obtaining the fine coordinate prediction results in the local relative coordinate system, based on the absolute offset recorded during the clipping operation ( , ) and scaling factor Perform an inverse affine transformation to map the fine coordinate prediction results back to the global coordinate system of the original full image to obtain the actual executable coordinates; Determine whether the current iteration step has reached the preset number of coarse-to-fine iteration focusing times N. If not, select the winning box of the current round as the new anchor point and return to perform the local clipping operation. If it has reached the target, extract the center point coordinates of the final winning box and map them back to the original image dimension as the final execution anchor point output of the graphical user interface agent.

4. The method according to claim 1, characterized in that, The process involves applying a dynamic mask to the input image of the current round, obscuring previously generated historical candidate regions, and then repeatedly predicting the masked image against the natural language command input to the basic localization model. This generates multiple spatially exclusive candidate box sets, including: Initialize the candidate set for the current round to an empty set, enter the candidate generation inner loop and set the loop variable. For each loop, call the mask function to reset the pixel values ​​of all historical candidate box regions that already exist in the candidate set in the current input image to zero vectors to generate pure black occlusion areas, thereby obtaining the mask image. The natural language instructions and the mask image are re-input into the basic localization model for inference calculation to obtain new candidate boxes and add them to the candidate set, ensuring that the newly generated candidate boxes do not overlap with existing erroneous candidates; Repeat the mask image construction and re-prediction accumulation steps until the loop ends to obtain a specified number of candidate boxes that are mutually exclusive in spatial distribution and semantically discriminative, in order to overcome the ambiguity bias caused by the multimodal model ignoring the actual physical space Euclidean distance.

5. The method according to claim 1, characterized in that, The process involves constructing a comparison image containing visual annotation information for each candidate box in the candidate box set, inputting the comparison image, natural language instructions, and preset selection rule prompts into the error correction selection model, and determining the optimal candidate box from the candidate box set as the final localization result through comparison and evaluation. This includes: On the original image copy, each candidate box in the candidate box set is explicitly rendered using different geometric color markers and numerical numbers, generating a multi-candidate comparison image with labeled visual cues to provide an intuitive spatial comparison basis; Construct structured prompts that include mandatory interface selection principles and thought chains. The selection principles stipulate that functional attribute matching takes priority over other attributes, interactive controls take priority over static text components, and define the interface element quality hierarchy as follows: icon combined with text area is better than pure complete icon, pure complete icon is better than pure complete text, and pure complete text is better than incomplete area of ​​multiple elements stacked. The natural language instruction, the multi-candidate comparison image, and the structured prompt words are input into an error-correcting selection model with visual question-answering capabilities. The error-correcting selection model calculates the posterior preference probability of each candidate box based on the evaluation weight of the prompt words, and outputs the winning candidate box identifier that maximizes the probability as the final localization result after deambiguity biasing.

6. A device for improving GUI localization capabilities without training based on operation chains, characterized in that, include: The prediction module is used to acquire the original image and natural language command of the target interface, input the original image and natural language command into the basic localization model for global prediction, and parse the output results to obtain the initial candidate region. The mapping module is used to perform local cropping operations on the original image based on the initial candidate region to generate a local sub-image, perform a second prediction in the coordinate system of the local sub-image to obtain a fine localization result, and then backmap the fine localization result back to the global coordinate system of the original image to update the candidate region. The masking module is used to apply dynamic masking to the input image of the current round, occluding the generated historical candidate regions. The masked image and natural language instructions are then input into the basic localization model for repeated prediction, generating multiple sets of candidate boxes that are mutually exclusive in spatial distribution. The evaluation module is used to construct a comparison image containing visual annotation information of each candidate box in the candidate box set. The comparison image, natural language instructions, and preset selection rule prompts are input into the error correction selection model. The optimal candidate box is determined from the candidate box set as the final localization result through comparison and evaluation.

7. An electronic device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the method as described in any one of claims 1-5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-5.

Images