Graphical user interface positioning method and device, computer equipment and storage medium

Abstract

The invention provides a graphical user interface positioning method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring an image of a graphical user interface and a natural language instruction; superposing a visually visible first grid and corresponding identification information for the image of the graphical user interface to obtain a first to-be-positioned image; inputting the first to-be-positioned image and the natural language instruction into a pre-trained visual language model to obtain identification information of a target first grid corresponding to a target element associated with the natural language instruction; intercepting a target area corresponding to the target element from the image according to the identification information of the target first grid, and forming a second to-be-positioned image based on the target area; inputting the second to-be-positioned image, the first to-be-positioned image and the natural language instruction into a visual language model to obtain identification information of a target second grid corresponding to the target element; and determining position information of the target element in the graphical user interface according to the identification information of the target second grid.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a positioning method, apparatus, computer device, and storage medium for a graphical user interface. Background Technology

[0002] A graphical user interface (GUI) is an interface design that allows users to interact with electronic devices, such as computers, mobile phones, and tablets, through graphical elements. It uses visual icons, buttons, windows, menus, etc., to replace complex code or instructions, making operation more intuitive.

[0003] With the significant breakthroughs achieved by large-scale visual language models (VLMs) in tasks such as image description, visual question answering, and text-to-image processing, using VLMs for GUI operations has also become a widely used technique. Existing VLMs typically output localization results using continuous coordinates or bounding box regression. When small targets or overlapping controls exist in the GUI interface, coordinate prediction is prone to jitter or offset, affecting localization accuracy. Summary of the Invention

[0004] In view of this, this application provides a positioning method, apparatus, computer device, and storage medium for a graphical user interface.

[0005] Specifically, this application is implemented through the following technical solution: In a first aspect, embodiments of this disclosure provide a method for locating a graphical user interface, the method comprising: Acquire images of the graphical user interface and natural language instructions for interacting with the graphical user interface. A first visible grid and the identification information of each first grid are overlaid on the image of the graphical user interface to obtain a first image to be located; The first image to be located and the natural language instruction are input into a pre-trained visual language model to obtain the identification information of the first grid corresponding to the target element associated with the natural language instruction; Based on the identification information of the first target grid, a target region corresponding to the target element is extracted from the image, and a second image to be located is constructed based on the target region; wherein, the second image to be located includes an image obtained by magnifying the target region; The second image to be located, the first image to be located, and the natural language instruction are input into the visual language model to obtain the identification information of the target second grid corresponding to the target element; and the position information of the target element in the graphical user interface is determined based on the identification information of the target second grid.

[0006] Optionally, overlaying the first grid onto the image of the graphical user interface includes: Determine the grid size corresponding to the graphical user interface; The first grid is overlaid on the image of the graphical user interface according to the grid size.

[0007] Optionally, determining the grid size corresponding to the graphical user interface includes: The grid size is determined based on the attribute information of each element in the graphical user interface; the attribute information includes at least one of the following: type, size, quantity, and deployment concentration. The attribute information is determined by reading the code of the graphical user interface; or it is obtained by performing element recognition processing on the image of the graphical user interface.

[0008] Optionally, determining the grid size corresponding to the graphical user interface includes: The grid size corresponding to the graphical user interface is determined based on the pre-set arrangement of the first grid and the size of the graphics in the graphical user interface.

[0009] Optionally, constructing a second image to be located based on the target region includes: The target area is magnified; The second image to be located is obtained by overlaying a visually visible second grid and the identification information of each second grid onto the magnified target area.

[0010] Optionally, determining the position information of the target element in the graphical user interface based on the identification information of the target second grid includes: Based on the pre-established positional relationship between the center of each second grid and the second image to be located, the center point position of each target second grid is determined; Based on the center point position of each target's second grid, determine the center coordinates or bounding box position coordinates of the target element; The position information of the target element in the graphical user interface is determined based on the center coordinates or bounding box position coordinates of the target element.

[0011] Optionally, the method further includes: In the first image to be located, the target first grid is marked; The step of inputting the second image to be located, the first image to be located, and the natural language instruction into the visual language model to obtain the identification information of the target second grid corresponding to the target element includes: The second image to be located, the first image to be located after marking the first grid of the target, and the natural language instruction are input into the visual language model to obtain the identification information of the second grid of the target element.

[0012] Optionally, it also includes: performing an operation corresponding to the natural language instruction on the target element based on the position information of the target element in the graphical user interface.

[0013] Secondly, embodiments of this disclosure also provide a positioning device for a graphical user interface, the device comprising: The acquisition module is used to acquire images of the graphical user interface and natural language instructions for interacting with the graphical user interface; The processing module is used to overlay a visually visible first grid and the identification information of each first grid onto the image of the graphical user interface to obtain a first image to be located; The prediction module is used to input the first image to be located and the natural language instruction into a pre-trained visual language model to obtain the identification information of the target first grid corresponding to the target element associated with the natural language instruction; The processing module is further configured to extract a target region corresponding to the target element from the image based on the identification information of the first target grid, and construct a second image to be located based on the target region; wherein the second image to be located includes an image obtained by magnifying the target region; The prediction module is further configured to input the second image to be located, the first image to be located, and the natural language instruction into the visual language model to obtain the identification information of the target second grid corresponding to the target element; and to determine the position information of the target element in the graphical user interface based on the identification information of the target second grid.

[0014] Optionally, the processing module, when overlaying the first grid onto the image of the graphical user interface, is used to: Determine the grid size corresponding to the graphical user interface; The first grid is overlaid on the image of the graphical user interface according to the grid size.

[0015] Optionally, the processing module, when determining the grid size corresponding to the graphical user interface, is used to: The grid size is determined based on the attribute information of each element in the graphical user interface; the attribute information includes at least one of the following: type, size, quantity, and deployment concentration. The attribute information is determined by reading the code of the graphical user interface; or it is obtained by performing element recognition processing on the image of the graphical user interface.

[0016] Optionally, the processing module, when determining the grid size corresponding to the graphical user interface, is used to: The grid size corresponding to the graphical user interface is determined based on the pre-set arrangement of the first grid and the size of the graphics in the graphical user interface.

[0017] Optionally, the processing module, when constructing a second image to be located based on the target region, is used to: The target area is magnified; The second image to be located is obtained by overlaying a visually visible second grid and the identification information of each second grid onto the magnified target area.

[0018] Optionally, the prediction module, when determining the position information of the target element in the graphical user interface based on the identification information of the target second grid, is used to: Based on the pre-established positional relationship between the center of each second grid and the second image to be located, the center point position of each target second grid is determined; Based on the center point position of each target's second grid, determine the center coordinates or bounding box position coordinates of the target element; The position information of the target element in the graphical user interface is determined based on the center coordinates or bounding box position coordinates of the target element.

[0019] Optionally, the processing module is further configured to: mark the first grid of the target in the first image to be located; The prediction module is specifically used to: input the second image to be located, the first image to be located after marking the first grid of the target, and the natural language instruction into the visual language model to obtain the identification information of the second grid of the target element.

[0020] Optionally, it also includes an execution module, configured to: perform an operation on the target element corresponding to the natural language instruction based on the position information of the target element in the graphical user interface.

[0021] Thirdly, an optional implementation of this disclosure also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the first aspect above, or any possible implementation of the first aspect.

[0022] Fourthly, an optional implementation of this disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the first aspect or any possible implementation of the first aspect.

[0023] Fifthly, an optional implementation of this disclosure also provides a computer program product, the computer program product carrying program code, the program code including instructions that can be used to perform the steps of the specific embodiments as described in the first aspect or any one of the first aspects.

[0024] This disclosure provides a graphical user interface (GUI) that labels images with grids and corresponding grid identification information. Using the grids and grid identification information as references, each grid has unique identification information. The visual language model only needs to predict the grid identification information where the boundary of the target element is located, thereby significantly reducing coordinate jitter. Furthermore, through a multi-level scaling method of grid labeling, multi-level localization of target elements in the GUI is performed, enabling the visual language model to achieve high-precision pixel-level localization without retraining.

[0025] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0026] Figure 1 A flowchart illustrating a positioning method for a graphical user interface provided in some embodiments of this disclosure is shown; Figure 2 The present disclosure shows a schematic diagram of the structure of a computer device provided in some embodiments; Figure 3 A schematic diagram of the structure of a positioning device for a graphical user interface provided in some embodiments of this disclosure is shown. Detailed Implementation

[0027] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0028] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0029] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0030] With the development of Vision-Language Models (VLMs), visual language understanding has expanded from natural image scenes to graphical user interface (GUI) tasks. GUI grounding is an important direction in this field, aiming to accurately locate target elements in the GUI interface based on natural language instructions, providing fundamental capabilities for applications such as intelligent agent interaction, automated testing, and software operation.

[0031] Currently, to achieve grounding capabilities of visual language models in GUI interfaces, it is typically necessary to rely on large-scale labeled data or model fine-tuning, achieving grounding by explicitly learning the correspondence between interface elements and text descriptions. For example, UGround uses approximately 1.3 million interface screenshots and 10 million element-labeled samples, AGUVIS relies on 4.2 million screenshots and 1.3 million manually generated trajectory samples, and other models such as CogAgent, OS-ATLAS, and OmniParser also require massive labeled corpora and task-specific training. Although these methods achieve good accuracy, their training costs are high, their transferability is poor, and they are difficult to apply quickly to new scenarios. Furthermore, GUI interfaces have characteristics significantly different from natural images: dense elements, regular structures, and large differences in target size. Traditional VLMs often struggle to accurately identify the spatial positions of interface elements in such tasks, leading to large grounding errors.

[0032] Furthermore, current methods often use continuous coordinates or bounding box regression to output positioning results; when there are small targets or overlapping controls in the GUI, coordinate prediction is prone to jitter or offset, affecting positioning accuracy.

[0033] Therefore, researching how to enhance the spatial awareness and localization capabilities of multimodal models in GUI tasks without retraining has become an urgent problem to be solved.

[0034] To address the aforementioned issues, this disclosure provides a graphical user interface (GUI) that labels images with grids and corresponding grid identification information. Using the grids and grid identification information as references, each grid has a unique identifier. The visual language model only needs to predict the grid identification information of the target element's boundary, thus significantly reducing coordinate jitter. Furthermore, through a multi-level scaling method for grid labeling, multi-level localization of target elements in the GUI is achieved, enabling the visual language model to achieve high-precision pixel-level localization without retraining.

[0035] The shortcomings of the above solutions are the result of the inventor's practical experience and careful research. Therefore, the discovery process of the above problems and the solutions proposed in this disclosure below should be considered as the inventor's contribution to this disclosure.

[0036] To facilitate understanding of the technical solutions disclosed herein, the technical terms used in the embodiments of this disclosure will first be explained: Vision-Language Models (VLMs) are a class of multimodal AI models that integrate visual information processing with natural language understanding / generation capabilities. Their core objective is to break down the modal barriers between visual data (images, videos, etc.) and natural language data (text, speech, etc.), enabling cross-modal semantic alignment, interaction, and reasoning. The essence of VLMs is establishing a mapping relationship between visual features and linguistic features, deeply fusing information from both modalities through a model architecture, ultimately possessing three core capabilities: Cross-modal understanding: Simultaneously analyzing the semantic relationships between visual and linguistic information (such as determining whether "there is a red cup in the picture" or "whether the text description matches the image"). Cross-modal generation: generating language content (such as image descriptions and video summaries) based on visual input, or editing visual content (such as modifying image elements based on text) based on language instructions; Cross-modal reasoning: Completing complex logical tasks based on the combination of vision and language (such as "answering mathematical questions based on chart data" or "analyzing the causal relationships of events in a video").

[0037] The core of using VLM for GUI operations is to rely on VLM's cross-modal understanding capabilities, combined with dedicated tools or customized training schemes, to enable the model to map natural language commands to visual elements of the GUI interface, locate the position of the target visual element in the GUI, and then use the location results to complete operations such as clicking and input.

[0038] To facilitate understanding of this embodiment, a detailed description of the graphical user interface (GUI) positioning method disclosed in this disclosure is provided first. The executor of the GUI positioning method provided in this disclosure is generally a computer device with certain computing capabilities. This computer device may include, for example, a terminal device, a server, or other processing devices. The terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, an in-vehicle device, a wearable device, etc. In some possible implementations, the GUI positioning method can be implemented by a processor calling computer-readable instructions stored in memory.

[0039] The positioning method of the graphical user interface provided in the embodiments of this disclosure will be described below.

[0040] See Figure 1 The diagram shows a flowchart of a graphical user interface positioning method provided in this embodiment of the present disclosure. The method includes steps S101 to S104, wherein: S101: Obtain an image of the graphical user interface and natural language instructions for interacting with the graphical user interface.

[0041] In practice, the image of the graphical user interface (GUI) can be acquired, for example, after the computer device's interface transitions to the GUI. The GUI image can be automatically acquired each time an interface transition event occurs.

[0042] In addition, the image of the graphical user interface can also be acquired only after a specific event is triggered. For example, some graphical user interfaces include controls for user input. After the user enters information into the control, if the user needs to perform other operations on the control, such as deleting the information entered into the control, the image of the graphical user in the current state can be acquired in real time when the specific event is triggered by an instruction.

[0043] Natural language commands used to interact with graphical user interfaces can be in text or voice form.

[0044] For natural language commands in speech form, corresponding speech recognition models can be used to convert the speech commands into text.

[0045] Natural language instructions are first converted into an encoding pattern before being input into the visual language model, so as to facilitate subsequent processing by the visual language model.

[0046] S102: Overlay a visually visible first grid and the identification information of each of the first grids onto the image of the graphical user interface to obtain a first image to be located.

[0047] In practice, the grid size of the first grid superimposed on the original image can be, for example, pre-set or determined in real time according to the actual situation of the GUI.

[0048] In addition, the layout of the first grid can be set, and the grid size of the first grid can be determined based on the layout and the resolution of the original image. Then, the first grid is overlaid on the original image. For example, the grid layout can be an 8×8 grid.

[0049] For example, a first grid can be overlaid on the image of the graphical user interface in the following manner: The size of the first grid is determined based on at least one of the following: the type, size, quantity, and deployment concentration of each element in the graphical user interface; The first grid is overlaid on the image of the graphical user interface according to the grid size.

[0050] Specifically, the types of elements in a graphical user interface can include, for example: buttons, hyperlinks, icon buttons, windows, panels, tabs, images, icons, lists, tables, progress bars, text input boxes, checkboxes, radio buttons, drop-down menus, sliders, date pickers, etc.

[0051] Different elements typically have different size ranges. For example, buttons, icons, and other elements are usually smaller. If you want to manipulate these elements, you can set a smaller grid size for them.

[0052] For elements such as windows, lists, and text input boxes, which are usually quite large, you can set a larger grid size if you want to manipulate these elements.

[0053] Furthermore, elements of the same type may have different sizes, so the size of the first grid can be determined based on the size distribution of the elements.

[0054] In addition, the more elements there are in the GUI, the smaller the size can be set to detect the target elements more accurately.

[0055] The more concentrated the elements are deployed, the smaller the size can be to detect the target elements more accurately.

[0056] The specific size of the first grid can be set according to actual needs, and this embodiment does not limit it.

[0057] The information regarding the type, size, quantity, and deployment concentration of each element in the graphical user interface can be obtained by recognizing the GUI image using a target recognition model, or by reading and analyzing the graphical user interface code.

[0058] Furthermore, the layout of a graphical user interface (GUI) for a specific application is usually relatively fixed. Therefore, the above method can be used to determine the size of the first grid for a GUI, and then use this size as the association information of the GUI, storing it in association with the GUI's identification information. When the GUI needs to be positioned, the size of the corresponding first grid can be directly read from the relevant storage location, and the read size can be used to directly overlay the first grid onto the GUI.

[0059] In this embodiment of the disclosure, the first grid superimposed on the graphical user interface and the identification information added to each first grid are visually explicit information, that is, as content in the image, are added to the image of the graphical user interface to obtain the first image to be located.

[0060] Following S102 above, the graphical user interface positioning method provided in this embodiment of the disclosure further includes: S103: Input the first image to be located and the natural language instruction into a pre-trained visual language model to obtain the identification information of the target first grid corresponding to the target element associated with the natural language instruction.

[0061] In practice, the first image to be localized is input into a pre-trained visual language model, which can extract high-dimensional features from the first image to be localized in order to project the image into a high-dimensional visual space.

[0062] When a natural language instruction is input into a visual language model, the visual language model can extract high-dimensional features from the natural language instruction and project the natural language instruction into a high-dimensional language space.

[0063] Then, cross-modal matching is performed between the first image to be localized and the natural language instruction to locate the target element related to the natural language instruction from the first image to be localized.

[0064] In this embodiment of the disclosure, since a first grid and visually recognizable identifiers of each first grid are superimposed on the image of the graphical user interface, the visual language model can identify each grid and its corresponding identifiers. Therefore, after inputting the first image to be located and the natural language command into the pre-trained visual language model, the visual language model can output the target first grid corresponding to the target element associated with the natural language, and output the identifier information of the target first grid.

[0065] In one alternative implementation, the visual language model can identify the location information of the target element in the image, which includes, for example, the relevant coordinate values ​​of the bounding box. Furthermore, since visually visible first grids and corresponding labeling information are added to each first grid, and the visual language model can recognize this labeling information, the labeling information of the first grid in the image where the target element is located can be determined based on the location information corresponding to the target element and the identified labeling information.

[0066] The bounding box information includes, for example, the coordinates of the target vertices in the image coordinate system, as well as the height and width of the bounding box. Based on this height and width information, the coordinates of the four vertices corresponding to the bounding box can be determined. The first grid cell into which the coordinates of these four vertices fall is designated as the target first grid cell, and the identification information identified from the image region containing the first grid cell is used as the identification information of the target element within the target first grid cell in the image.

[0067] Furthermore, in another embodiment of this disclosure, the identification information of the first grid where the four boundaries of the target element are located can also be directly predicted using a visual language model, such as the identification information of the target first grid corresponding to the upper boundary, lower boundary, left boundary, and right boundary, respectively. Alternatively, the identification information of the first grid where the four vertices of the target element are located can be directly predicted, such as the identification information of the target first grid corresponding to the upper left vertex, upper right vertex, lower left vertex, and lower right vertex, respectively.

[0068] Then, based on the identification information of the four boundaries of the predicted target element in the first grid, or based on the identification information of the four vertices of the predicted target element in the first grid, the identification information of the target first grid in the image is determined.

[0069] For example, suppose that when adding tags to each first grid, they are added in row and column order. Suppose that the first grid has m rows and n columns. For example, the tag information of the first grid in the first row and first column is 1-1, the tag information of the first grid in the first row and second column is 1-2, ..., and the tag information of the first grid in the first row and nth column is 1-n.

[0070] The identifier for the first grid in the first column of the second row is 2-1; the identifier for the first grid in the second column of the second row is 2-2; ...; the identifier for the first grid in the nth column of the second row is 2-n.

[0071] ... The identifier of the first grid in the first column of the m-th row is m-1, the identifier of the first grid in the second column of the m-th row is m-2, ..., the identifier of the first grid in the n-th column of the m-th row is mn.

[0072] Assuming the target element has four boundaries, the identifier of the first grid cell containing the left boundary is k1-j1, the identifier of the first grid cell containing the right boundary is k1-j2 (the left and right boundaries have the same rows but different columns), the identifier of the first grid cell containing the top boundary is k2-j3, and the identifier of the first grid cell containing the bottom boundary is k3-j3 (the top and bottom boundaries have different rows but the same columns). Typically, k3 ≥ k1 ≥ k2, and j2 ≥ j3 ≥ j1. Therefore, the row values ​​in the identifier of the first grid cell are in the range [k2, k3], and the column values ​​are in the range [k1, k2]. Based on these ranges, the identifier of the first grid cell corresponding to the target element can be determined.

[0073] S104: Based on the identification information of the first target grid, a target region corresponding to the target element is extracted from the image, and a second image to be located is constructed based on the target region; wherein, the second image to be located includes an image obtained by magnifying the target region.

[0074] In practical implementation, after obtaining the identification information of the target grid, the target region corresponding to the target element can be extracted from the image of the GUI based on the identification information of the first target grid. Since the target region is determined using grids, rather than relying entirely on the prediction of the visual language model, even if the predicted bounding box is offset, it will be corrected by the first grid.

[0075] Next, the target region corresponding to the target element is extracted from the image, and a second image to be located is constructed based on this target region. This second image to be located includes an image obtained by magnifying the target region.

[0076] Specifically, when extracting the target region corresponding to the target element from the image based on the identification information of the target first grid, since the identification information of the obtained target off-site grid includes the target first grid corresponding to the four corners or four sides respectively, the region where each target first grid is located can be deduced based on the identification information of the target first grid, and the region can be extracted to obtain the target region.

[0077] In one possible implementation, when constructing a second image to be located based on the target region, the target region can be enlarged to the target size; in addition, the length and width of the target region can be enlarged, and then the enlarged target region can be scaled proportionally to the target size (e.g., 512 pixels) to obtain the second image to be located.

[0078] Then, the obtained second image to be located, the first image to be located, and the natural language instruction are input into the visual language model to obtain the position information of the target element in the second image to be located.

[0079] Subsequently, based on the magnification factor of the second image to be located and the position information, the position information of the target element determined based on the second image to be located is projected onto the image of the graphical user interface to obtain the position information of the target element in the image of the graphical user interface.

[0080] In another possible implementation, the target region is magnified; The second image to be located is obtained by overlaying a visually visible second grid and the identification information of each second grid onto the magnified target area.

[0081] Here, when overlaying a visually visible second grid onto the magnified target area, the overlay method is similar to that used for overlaying a first grid onto an image in a graphical user interface. The second grid can be overlaid onto the magnified target area according to a specific grid size. Furthermore, the grid size of the second grid can be the same as or different from the grid size of the first grid; specific embodiments of this disclosure do not limit this.

[0082] S105: Input the second image to be located, the first image to be located, and the natural language instruction into the visual language model to obtain the identification information of the target second grid corresponding to the target element; and determine the position information of the target element in the graphical user interface based on the identification information of the target second grid.

[0083] In a specific implementation, after inputting the second image to be located, the image of the graphical user interface, and the natural language command into the visual language model, the image of the graphical user interface and the natural language command are used as references to identify the target element to be identified from the second image to be located, thereby obtaining the identification information of the second grid corresponding to the target element in the second image to be located.

[0084] Then, the location information of the target element in the graphical user interface can be determined based on the identification information of the second grid of the target.

[0085] In another embodiment of this disclosure, for example, the target first grid may also be marked in the first image to be located.

[0086] Then, the second image to be located, the first image to be located after the target first grid has been marked, and the natural language instruction are input into the visual language model to obtain the identification information of the target second grid corresponding to the target element.

[0087] Here, the target first grid is marked, for example, the lines of the target first grid can be thickened, the color can be adjusted, or text can be marked.

[0088] In this way, by marking the first target grid, the visual language model can intuitively learn the relative positional relationship between the second image to be processed and the first image to be processed, thereby enhancing the reference capability of the first image to be processed and improving the recognition accuracy of the identification information of the target element corresponding to the target second micro-grid.

[0089] Specifically, when determining the position information of the target element in the graphical user interface, the following methods can be used, for example: Based on the pre-established positional relationship between the center of each second grid and the second image to be located, the center point position of each target second grid is determined; Based on the center point position of each target's second grid, determine the center coordinates or bounding box position coordinates of the target element; The position information of the target element in the graphical user interface is determined based on the center coordinates or bounding box position coordinates of the target element.

[0090] Specifically, since elements in a graphical user interface come in various categories, and different categories correspond to different operations, the output position information can also differ depending on the type of target element. For example, for trigger-type elements, such as buttons and hyperlinks, since they are triggered by clicking, only the corresponding area of ​​the element needs to be clicked; therefore, the center coordinates of the target element can be output.

[0091] For operations triggered by other means, such as input operations, which usually operate on a certain area, the bounding box of the target element can be output.

[0092] In another embodiment of this disclosure, an operation corresponding to the natural language instruction is performed on the target element based on the position information of the target element in the graphical user interface.

[0093] Here, the operations corresponding to natural language are obtained through a visual language model. After the position of the target element is output through the visual language model, the target element can be triggered accordingly based on that position, and the operation corresponding to the natural language instruction can be executed.

[0094] The graphical user interface localization method provided in this disclosure transforms the problem of continuous coordinate prediction of visual language models in the GUI interface into a discrete classification task based on grid ID, and combines multi-level scaling inference to achieve fine-grained localization from global to local.

[0095] Unlike traditional continuous coordinate regression, this invention uses a grid superimposed on the image, with each grid cell having a unique number. The model only needs to predict the grid ID where the target boundary is located, thus significantly reducing coordinate prediction jitter. Furthermore, through a two-level inference structure of "coarse localization—cropping and magnification—fine localization," the model can achieve high-precision pixel-level localization without modifying the parameters.

[0096] This approach can improve the spatial understanding and localization accuracy of multimodal models in GUI interface tasks without additional training, and belongs to the input structure enhancement type of localization capability improvement method.

[0097] The graphical user interface positioning method provided in this disclosure has the following beneficial effects: 1. Positioning accuracy is significantly improved: By transforming continuous coordinate prediction into discrete grid label classification and combining it with a multi-level scaling inference mechanism, this method significantly reduces coordinate jitter and regression error, and can achieve higher-precision pixel-level positioning in GUI interface scenarios with dense small targets.

[0098] 2. No model training required, exhibiting zero-shot adaptability: This invention enhances localization capabilities through input augmentation, without requiring any parameter updates or task-specific fine-tuning of the multimodal model, and can be directly applied to different models and interface tasks.

[0099] 3. Improve the spatial understanding and stability of the model: By adopting a hierarchical reasoning approach from global to local, the model can maintain overall semantic understanding while possessing fine-grained localization capabilities, thereby improving the robustness and consistency of the model in complex interfaces.

[0100] 4. High versatility and easy integration: This method only depends on changes in the input structure and does not require modification of the model architecture. It can be directly embedded into existing VLM inference processes and is suitable for various scenarios such as GUI agents, automated testing, and interactive operations.

[0101] Corresponding to the aforementioned embodiments of the graphical user interface positioning method, this application also provides embodiments of a graphical user interface positioning device.

[0102] The positioning device embodiments of the graphical user interface of this application can be applied to computer devices. The device embodiments can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of the computer device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 2 The diagram shown is a hardware structure diagram of a computer device containing the positioning device of the graphical user interface of this application. Except for... Figure 2 In addition to the processor, memory, network interface, and non-volatile memory shown, the computer device in which the device is located in the embodiment may also include other hardware depending on the actual function of the positioning device of the graphical user interface, which will not be described in detail here.

[0103] Please refer to Figure 3 The positioning device for the graphical user interface provided in this embodiment includes: The acquisition module 31 is used to acquire images of the graphical user interface and natural language instructions for interacting with the graphical user interface; Processing module 32 is used to overlay a visually visible first grid and the identification information of each first grid onto the image of the graphical user interface to obtain a first image to be located; Prediction module 33 is used to input the first image to be located and the natural language instruction into a pre-trained visual language model to obtain the identification information of the target first grid corresponding to the target element associated with the natural language instruction; The processing module 32 is further configured to extract a target region corresponding to the target element from the image based on the identification information of the first target grid, and construct a second image to be located based on the target region; wherein the second image to be located includes an image obtained by magnifying the target region; The prediction module 33 is further configured to input the second image to be located, the first image to be located, and the natural language instruction into the visual language model to obtain the identification information of the target second grid corresponding to the target element; and to determine the position information of the target element in the graphical user interface based on the identification information of the target second grid.

[0104] Optionally, the processing module 32, when overlaying the first grid onto the image of the graphical user interface, is used to: Determine the grid size corresponding to the graphical user interface; The first grid is overlaid on the image of the graphical user interface according to the grid size.

[0105] Optionally, the processing module 32, when determining the grid size corresponding to the graphical user interface, is used to: The grid size is determined based on the attribute information of each element in the graphical user interface; the attribute information includes at least one of the following: type, size, quantity, and deployment concentration. The attribute information is determined by reading the code of the graphical user interface; or it is obtained by performing element recognition processing on the image of the graphical user interface.

[0106] Optionally, the processing module 32, when determining the grid size corresponding to the graphical user interface, is used for: The grid size corresponding to the graphical user interface is determined based on the pre-set arrangement of the first grid and the size of the graphics in the graphical user interface.

[0107] Optionally, the processing module 32, when extracting the target region corresponding to the target element from the image based on the identification information of the first target grid, and constructing a second image to be located based on the target region, is used to: The target area is magnified; The second image to be located is obtained by overlaying a visually visible second grid and the identification information of each second grid onto the magnified target area.

[0108] Optionally, the prediction module 33, when inputting the second image to be located into the visual language model to obtain the identification information of the target second grid corresponding to the target element, is used to: The image of the graphical user interface and the second image to be located are input into the visual language model, so as to use the image of the graphical user interface as a reference to identify the target element from the second image to be located, and obtain the identification information of the target second grid corresponding to the target element.

[0109] Optionally, the prediction module 33, when determining the position information of the target element in the graphical user interface based on the identification information of the target second grid, is used to: Based on the pre-established positional relationship between the center of each second grid and the second image to be located, the center point position of each target second grid is determined; Based on the center point position of each target's second grid, determine the center coordinates or bounding box position coordinates of the target element; The position information of the target element in the graphical user interface is determined based on the center coordinates or bounding box position coordinates of the target element.

[0110] Optionally, the processing module 32 is further configured to: mark the target first grid in the first image to be located; The prediction module 33 is specifically used to: input the second image to be located, the first image to be located after marking the first grid of the target, and the natural language instruction into the visual language model to obtain the identification information of the second grid of the target element.

[0111] Optionally, it also includes an execution module 34, configured to: perform an operation corresponding to the natural language instruction on the target element based on the position information of the target element in the graphical user interface.

[0112] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0113] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the graphical user interface positioning method described in the above-described method embodiments. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0114] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the graphical user interface positioning method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0115] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0116] The computer program or instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions may be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; or an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.

[0117] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0118] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A positioning method for a graphical user interface, characterized in that, The method includes: Acquire images of the graphical user interface and natural language instructions for interacting with the graphical user interface. A first visible grid and the identification information of each first grid are overlaid on the image of the graphical user interface to obtain a first image to be located; The first image to be located and the natural language instruction are input into a pre-trained visual language model to obtain the identification information of the first grid corresponding to the target element associated with the natural language instruction; Based on the identification information of the first target grid, a target region corresponding to the target element is extracted from the image, and a second image to be located is constructed based on the target region; wherein, the second image to be located includes an image obtained by magnifying the target region; The second image to be located, the first image to be located, and the natural language instruction are input into the visual language model to obtain the identification information of the target second grid corresponding to the target element; and the position information of the target element in the graphical user interface is determined based on the identification information of the target second grid.

2. The method according to claim 1, characterized in that, The process of overlaying a first grid onto the image of the graphical user interface includes: Determine the grid size corresponding to the graphical user interface; The first grid is overlaid on the image of the graphical user interface according to the grid size.

3. The method according to claim 2, characterized in that, Determining the grid size corresponding to the graphical user interface includes: The grid size is determined based on the attribute information of each element in the graphical user interface; the attribute information includes at least one of the following: type, size, quantity, and deployment concentration. The attribute information is determined by reading the code of the graphical user interface; or it is obtained by performing element recognition processing on the image of the graphical user interface.

4. The method according to claim 2, wherein determining the grid size corresponding to the graphical user interface includes: The grid size corresponding to the graphical user interface is determined based on the pre-set arrangement of the first grid and the size of the graphics in the graphical user interface.

5. The method according to claim 1, characterized in that, The process of constructing a second image to be located based on the target region includes: The target area is magnified; The second image to be located is obtained by overlaying a visually visible second grid and the identification information of each second grid onto the magnified target area.

6. The method according to claim 1, characterized in that, Determining the position information of the target element in the graphical user interface based on the identification information of the target second grid includes: Based on the pre-established positional relationship between the center of each second grid and the second image to be located, the center point position of each target second grid is determined; Based on the center point position of each target's second grid, determine the center coordinates or bounding box position coordinates of the target element; The position information of the target element in the graphical user interface is determined based on the center coordinates or bounding box position coordinates of the target element.

7. The method according to claim 1, characterized in that, The method further includes: In the first image to be located, the target first grid is marked; The step of inputting the second image to be located, the first image to be located, and the natural language instruction into the visual language model to obtain the identification information of the target second grid corresponding to the target element includes: The second image to be located, the first image to be located after marking the first grid of the target, and the natural language instruction are input into the visual language model to obtain the identification information of the second grid of the target element.

8. The method according to claim 1, characterized in that, Also includes: Based on the location information of the target element in the graphical user interface, perform the operation corresponding to the natural language instruction on the target element.

9. A positioning device for a graphical user interface, characterized in that, The device includes: The acquisition module is used to acquire images of the graphical user interface and natural language instructions for interacting with the graphical user interface; The processing module is used to overlay a visually visible first grid and the identification information of each first grid onto the image of the graphical user interface to obtain a first image to be located; The prediction module is used to input the first image to be located and the natural language instruction into a pre-trained visual language model to obtain the identification information of the target first grid corresponding to the target element associated with the natural language instruction; The processing module is further configured to extract a target region corresponding to the target element from the image based on the identification information of the first target grid, and construct a second image to be located based on the target region; wherein the second image to be located includes an image obtained by magnifying the target region; The prediction module is further configured to input the second image to be located, the first image to be located, and the natural language instruction into the visual language model to obtain the identification information of the target second grid corresponding to the target element; and to determine the position information of the target element in the graphical user interface based on the identification information of the target second grid.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1-8.

11. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1-8.

Images