A gui positioning method and system based on a routing prediction framework

Abstract

The application belongs to the technical field of computers and discloses a GUI positioning method and system based on a routing prediction framework. The application introduces a routing prediction mechanism based on word type discrimination in the autoregressive decoding process of a multi-modal large language model, dynamically shunts the interface semantic generation task and the spatial positioning task, enables the user instruction with the target element to jump out of the discrete text generation path, directly performs continuous spatial coordinate regression based on the hidden state and the interface image features enhanced by the structure, and returns an empty positioning result through a special rejection response word when the target element does not exist, thereby effectively avoiding the coordinate quantization error and the illusion positioning problem under the end-to-end unified framework, significantly reducing the positioning reasoning delay, improving the accuracy, real-time performance and robustness of the graphical user interface element positioning, and enhancing the overall understanding ability of the model to the interface structure semantics and spatial layout relationship.

Description

Technical Field

[0001] This invention relates to the field of computer technology, specifically to a GUI localization method and system based on a route prediction framework. Background Technology

[0002] With the development of Multimodal Large Language Models (MLLMs), they have shown potential in understanding and interacting with Graphical User Interfaces (GUIs), making it possible to build automated GUI agents. However, current MLLM-based GUI localization methods mainly follow the paradigm of processing natural images. When facing the highly structured, semantically dense, and spatially demanding specific domain of GUIs, they expose many technical shortcomings, limiting their practicality and implementation capabilities.

[0003] Existing technical solutions mainly fall into two categories. The first category is instruction-following fine-tuning based on open-source MLLM. This method uses a general MLLM as a foundation, fine-tunes it on a GUI positioning dataset, and outputs normalized coordinate values ​​by generating text token sequences through autoregression. The second category is prompt word engineering routes that directly call closed-source large model APIs. Through carefully designed system prompts and few-sample examples, it guides the model to directly output coordinate strings.

[0004] However, the above-mentioned existing technologies have the following common drawbacks: (1) Low positioning accuracy: Since the coordinate values ​​must be generated one by one through the text tokens discretized by the model vocabulary, this process introduces inherent quantization error and randomness of order, resulting in inaccurate predicted bounding box positions, which can easily lead to accidental touches in actual interaction.

[0005] (2) Slow inference speed: Generating a bounding box coordinate requires continuous autoregressive output of dozens of text tokens, which significantly increases the inference delay and makes it difficult to meet the application scenario requirements that require real-time response (such as mobile interaction delay of less than 200 milliseconds).

[0006] (3) Lack of ability to reject illusions: The vocabulary of the existing model does not have a dedicated token designed to indicate "target does not exist". When the user command requests to locate an element that does not exist in the interface, the model is forced to output a "valid" coordinate string, generating an "illusory" location box. This requires the downstream system to add an additional verification module for secondary processing, which increases the system complexity and maintenance cost.

[0007] (4) Insufficient understanding of structure: Existing methods simply regard GUI elements as objects or textures in natural images, failing to systematically model the strict semantic relationships and spatial layout relationships between interface elements, resulting in a weak understanding of the overall structure of the interface.

[0008] Therefore, there is an urgent need for an end-to-end solution that can overcome the above-mentioned defects and achieve high precision, high efficiency and high robustness in GUI perception tasks. Summary of the Invention

[0009] In view of the above-mentioned shortcomings mentioned in the background art, the purpose of this invention is to provide a GUI positioning method and system based on a route prediction framework.

[0010] Specifically, the first aspect of the present invention provides a GUI localization method based on a route prediction framework, the method comprising the following steps: Step S1, acquiring an image of a graphical user interface and user command text, and performing preprocessing and encoding to obtain an image feature sequence and a text feature sequence respectively, and concatenating the image feature sequence and the text feature sequence to obtain a fused feature sequence.

[0011] Step S2: Input the fused feature sequence into the multimodal large language model to obtain the hidden state output by the multimodal large language model in the current decoding step.

[0012] Step S3: Based on the hidden state, the token type output in the current decoding step is determined by the token routing unit, including visual positioning token, rejection response token, and ordinary text token.

[0013] Step S4: If the token type is a visual positioning token, the coordinate decoder is triggered to fuse and decode the hidden state corresponding to the visual positioning token with the image features enhanced by the visual adapter, and output continuous spatial coordinate values ​​as the bounding box of the target element.

[0014] If the token type is a rejection response token, an empty location result is returned, and the location request is rejected.

[0015] If the token type is a plain text token, then the multimodal large language model continues to autoregressively generate semantic description text.

[0016] A second aspect of the present invention provides a GUI positioning system based on a route prediction framework. The system includes an input acquisition and encoding module, which acquires images of a graphical user interface and user command text, performs preprocessing and encoding to obtain image feature sequences and text feature sequences respectively, and concatenates the image feature sequences and text feature sequences to obtain a fused feature sequence.

[0017] The multimodal inference module is used to input the fused feature sequence into the multimodal large language model and obtain the hidden state output by the multimodal large language model in the current decoding step.

[0018] The token routing module is used to determine the type of token output in the current decoding step based on the hidden state, including visual positioning tokens, rejection response tokens, and ordinary text tokens.

[0019] The positioning decoding module is used to trigger the coordinate decoder if the token type is a visual positioning token, to fuse and decode the hidden state corresponding to the visual positioning token with the image features enhanced by the visual adapter, and output continuous spatial coordinate values ​​as the bounding box of the target element.

[0020] If the token type is a rejection response token, an empty location result is returned, and the location request is rejected.

[0021] If the token type is a plain text token, then the multimodal large language model continues to autoregressively generate semantic description text.

[0022] A third aspect of the invention also provides a medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the preceding claims.

[0023] A fourth aspect of the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the method as described in any of the preceding claims.

[0024] Compared with existing technologies, the present invention has the following significant advantages: By introducing a route prediction mechanism based on term type discrimination in the autoregressive decoding process of a multimodal large language model, the present invention dynamically separates the interface semantic generation task and the spatial positioning task, enabling user commands with target elements to break out of the discrete text generation path and directly perform continuous spatial coordinate regression based on the hidden state and structure-enhanced interface image features. At the same time, when the target element does not exist, an empty positioning result is returned through a dedicated rejection response term. Thus, under the end-to-end unified framework, coordinate quantization error and phantom positioning problems are effectively avoided, the positioning inference latency is significantly reduced, the accuracy, real-time performance and robustness of graphical user interface element positioning are improved, and the model's overall understanding of interface structural semantics and spatial layout relationships is enhanced. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating a GUI localization method based on a route prediction framework disclosed in an embodiment of the present invention.

[0026] Figure 2 This is a schematic diagram of the process for generating fusion feature sequences disclosed in an embodiment of the present invention.

[0027] Figure 3 This is a schematic diagram of the structure of the coordination submodule disclosed in an embodiment of the present invention.

[0028] Figure 4 This is a schematic diagram of the structure of a GUI positioning system based on a route prediction framework disclosed in an embodiment of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] Please see Figure 1 This invention provides a GUI localization method based on a route prediction framework. The method includes the following steps: Step S1, acquiring the image of the graphical user interface and the user command text, and performing preprocessing and encoding to obtain image feature sequences and text feature sequences respectively, and concatenating the image feature sequences and text feature sequences to obtain a fused feature sequence.

[0031] This step first involves acquiring an image of the graphical user interface and the corresponding user instruction text. The interface image is a screenshot containing standardized interface elements such as buttons, icons, and text areas, while the user instruction text is a natural language statement describing the location or function of the target interface element.

[0032] The interface image undergoes preprocessing operations, including size normalization and pixel value standardization, to meet the input requirements of the visual encoder. Subsequently, the preprocessed interface image is input into the visual encoder, which divides the image into multiple local regions and maps each region to a continuous image feature vector, thus forming a spatially ordered sequence of image features. This image feature sequence can be represented as: ,in This represents the image feature vector corresponding to the i-th local region in the graphical user interface.

[0033] Simultaneously, the user command text undergoes text preprocessing and encoding. Specifically, the user command text is split into multiple discrete text units, and each text unit is converted into a continuous text feature vector through embedding mapping, thereby forming a text feature sequence, which can be represented as: ,in This represents the semantic feature vector corresponding to the j-th text unit in the user instruction.

[0034] Please see Figure 2 After constructing the image feature sequence and the text feature sequence, the two are concatenated along the sequence dimension to obtain the fused feature sequence. Furthermore, the fused feature sequence is used as input to a multimodal large language model to achieve joint modeling of interface visual information and instruction semantic information.

[0035] Step S2: Input the fused feature sequence into the multimodal large language model to obtain the hidden state output by the multimodal large language model in the current decoding step.

[0036] In this step, the fused feature sequence obtained in step S1 is input into the multimodal large language model. The multimodal large language model uses a multi-layer attention mechanism to jointly model image features and text features, thereby establishing the correspondence between the semantics of user commands and the structure of the graphical user interface.

[0037] During the stepwise decoding process of the model, the multimodal large language model outputs a corresponding hidden state vector at each decoding step. The hidden state can be represented as: ,in This represents the decoding function of a multimodal large language model. Indicates the current decoding step. This indicates the output sequence that has been generated before the current decoding step.

[0038] It is understandable that the aforementioned hidden state is a high-dimensional feature representation of the current decoding position, which comprehensively reflects the semantic content of the user command, the visual structural information of the interface image, and the relationship between the two, and serves as an important input for subsequent token routing judgment and coordinate decoding.

[0039] Step S3: Based on the hidden state, the token type output in the current decoding step is determined by the token routing unit, including visual positioning token, rejection response token, and ordinary text token.

[0040] In this step, the token routing unit receives the hidden state obtained in step S2 and maps the hidden state through the output projection layer of the multimodal large language model to obtain the prediction result corresponding to the expanded vocabulary, which can be expressed as: ,in and This is used to output the parameters of the projection layer.

[0041] Furthermore, the predicted probability corresponding to each vocabulary item is obtained through a normalization function: .

[0042] The expanded vocabulary adds visual location terms and rejection response terms to the original plain text terms. The token routing unit determines the term type for the current decoding step based on the predicted probability results. When the prediction result corresponds to a visual location term, it means the model determines that a target element matching the user's command exists in the current interface; when the prediction result corresponds to a rejection response term, it means the model determines that no target element matching the command exists in the current interface; when the prediction result corresponds to a plain text term, it means the model continues the semantic text generation process.

[0043] The token routing mechanism based on the hidden state prediction results described above enables dynamic splitting of semantic generation tasks and interface localization tasks.

[0044] Step S4: If the token type is a visual positioning token, the coordinate decoder is triggered to fuse and decode the hidden state corresponding to the visual positioning token with the image features enhanced by the visual adapter, and output continuous spatial coordinate values ​​as the bounding box of the target element.

[0045] If the token type is a rejection response token, an empty location result is returned, and the location request is rejected.

[0046] If the token type is a plain text token, then the multimodal large language model continues to autoregressively generate semantic description text.

[0047] This step is used to perform interface positioning processing, rejection processing, or semantic text generation processing according to the term type determined in step S3.

[0048] Specifically, when the current term type is determined to be a visual location term, the coordinate decoder is triggered to execute the location decoding process. At this point, the hidden state corresponding to the visual location term is extracted as a query feature and fused with the image feature sequence enhanced by the visual adapter. The fusion process is implemented through a cross-attention mechanism, which can be represented as follows: ; where, query vector From hidden state The mapping yields the key vector. Sum value vector Enhanced image feature sequence Mapped to obtain, For feature dimensions.

[0049] After feature fusion is completed, the fused features are input into the regression head, and the bounding box coordinates of the target interface elements are directly output, which can be represented as: The four consecutive values ​​constitute the normalized bounding box coordinates of the target interface element in the graphical user interface.

[0050] When the current term type is determined to be a rejection response term, the coordinate decoding operation is not performed, and an empty positioning result is returned directly to reject the positioning request; when the current term type is determined to be a plain text term, the multimodal large language model continues to perform the stepwise text generation process and outputs the corresponding semantic description text.

[0051] Through the above processing method, this invention introduces a route prediction mechanism based on term type discrimination in the autoregressive decoding process of a multimodal large language model, dynamically separating the interface semantic generation task and the spatial localization task. This allows user commands with target elements to bypass the discrete text generation path and directly perform continuous spatial coordinate regression based on the hidden state and structure-enhanced interface image features. At the same time, when the target element does not exist, a dedicated rejection response term is used to return an empty localization result. Thus, under the end-to-end unified framework, coordinate quantization error and phantom localization problems are effectively avoided, localization inference latency is significantly reduced, the accuracy, real-time performance and robustness of graphical user interface element localization are improved, and the model's overall understanding of interface structural semantics and spatial layout relationships is enhanced.

[0052] As an example, the token routing unit is implemented by reusing the output projection layer of a multimodal large language model; wherein the original vocabulary of the output projection layer is expanded to include two new vocabulary entries corresponding to the visual positioning token and the rejection response token.

[0053] In the decoding phase, the multimodal large language model maps the hidden state of the current decoding step to a preset vocabulary space through an output projection layer to obtain the predicted probability distribution corresponding to each vocabulary item. In this embodiment, while keeping the structure of the output projection layer unchanged, its corresponding original vocabulary is expanded. Based on the original ordinary text vocabulary items, two new vocabulary items with clear functional semantics are added, corresponding to the visual positioning token and the rejection response token, respectively.

[0054] In this way, after the hidden state is mapped by the output projection layer, the multimodal large language model can not only predict ordinary text vocabulary items, but also directly predict whether to output a visual positioning token or a rejection response token. The token routing unit determines the processing path that should be triggered in the current decoding step based on the prediction results of the output projection layer, thereby realizing dynamic routing for interface positioning, rejection positioning, or semantic text generation.

[0055] Since the token routing unit reuses the original output projection structure of the multimodal large language model, there is no need to introduce an independent discriminant network or complex rule judgment logic. Therefore, it can reduce the system implementation cost, reduce the parameter scale and inference overhead, and at the same time ensure the consistency and stability of the token discrimination process and the overall semantic modeling process of the model.

[0056] As an example, the visual adapter is a lightweight neural network module whose input is the original image feature sequence output by the visual encoder of a multimodal large language model. It is processed through a structure containing at least one bidirectional self-attention layer to output enhanced image features for the edges, rectangular regions, and text layout of target elements in a graphical user interface.

[0057] In this embodiment, the visual adapter is used to perform structured enhancement on the raw image features output by the visual encoder of the multimodal large language model, so as to improve the model's ability to perceive regularized interface elements in the graphical user interface.

[0058] Specifically, after encoding the interface image, the visual encoder of the multimodal large language model outputs a set of original image feature sequences arranged in spatial order, which can be represented as: ,in This represents the original image feature vector corresponding to the i-th image region in the interface image. The feature vector mainly contains local texture information and high-level semantic information.

[0059] Since the original image features mentioned above are mainly for semantic modeling of natural images, they are difficult to fully characterize the regular edges, rectangular region outlines and row and column arrangements of text regions that are common in graphical user interfaces without explicit structural constraints. Therefore, a visual adapter module is introduced between the visual encoder and the coordinate decoder.

[0060] In this embodiment, the visual adapter includes at least one bidirectional self-attention layer for performing relational modeling of the original image feature sequence on a global scale.

[0061] Specifically, the original image feature sequence These are mapped to query vector, key vector, and value vector, respectively: ,in , , It is a linear mapping function.

[0062] Subsequently, the bidirectional self-attention layer simultaneously calculates attention weights in both the forward and backward directions of the feature sequence. This allows each image feature to perceive not only its preceding spatial region information but also its subsequent regional information, thereby establishing a global correlation covering both horizontal and vertical directions. The attention calculation process can be represented as follows: ;in, For feature dimension, This is the intermediate feature sequence after bidirectional self-attention processing.

[0063] Through the aforementioned bidirectional self-attention processing, image features at different spatial locations can interact globally, resulting in the explicit enhancement of the following interface structural features: the feature responses of adjacent image regions at their edges are correlated, thereby enhancing the ability to model the continuity of interface element edges; image features distributed in a rectangular pattern in space can form consistent attention clusters, which is beneficial for depicting the overall outline of rectangular interface elements such as buttons and input boxes; and image features corresponding to text regions form ordered attention associations in the horizontal and vertical directions, thereby highlighting the row and column layout structure of the text regions.

[0064] The enhanced image feature sequence obtained after processing by the bidirectional self-attention layer can be represented as: ,in While retaining the original visual semantic information, it further integrates interface structure and layout features.

[0065] Enhanced image feature sequence As the output of the visual adapter, it is input to the cross-attention module in the coordinate decoder, where it is fused across modally with the hidden state corresponding to the visual localization token. Since the enhanced image features have explicitly encoded structural information such as the edges, rectangular regions, and text layout of interface elements, the subsequent cross-attention module can more accurately focus on regions that conform to the interface structural features when performing semantic-visual alignment, thereby improving the stability and accuracy of the bounding box regression results.

[0066] Through the above specific processing, the visual adapter not only achieves lightweight re-encoding of the original image features, but also introduces a structured enhancement mechanism for the characteristics of the graphical user interface without significantly increasing the model parameter scale and computational complexity, making the multimodal large language model more suitable for performing high-precision and high-robust interface element localization tasks.

[0067] As an example, the coordinate decoder includes a cross-attention module and a regression head; the cross-attention module uses the hidden state corresponding to the visual positioning token as the query vector and the enhanced image features as the key vector and value vector for feature fusion; the regression head is used to map the fused features into four consecutive floating-point numbers, which directly constitute normalized bounding box coordinates.

[0068] In this implementation, the coordinate decoder is used to deeply fuse the semantic hidden state output by the multimodal large language model with the image features enhanced by the visual adapter when the visual localization path is triggered, and outputs the spatial location information of the target interface element.

[0069] Specifically, the coordinate decoder is used to deeply fuse the semantic hidden state output by the multimodal large language model with the image features enhanced by the visual adapter when the visual localization path is triggered, and directly outputs the spatial location information of the target interface element. The coordinate decoder includes a cross-attention module and a regression head, which work together in sequence to complete the end-to-end mapping from semantic query to spatial coordinates.

[0070] When the token routing unit determines that the output of the current decoding step is a visual positioning token, it enters the visual positioning decoding path. At this time, the hidden state corresponding to the visual positioning token is extracted from the multimodal large language model, which can be represented as: ,in This is the hidden state vector for the current decoding step, used to represent the semantic constraints, functional descriptions, and localization intent of the user command regarding the target interface elements. The hidden state, as a high-level semantic representation of the localization query, is input into the cross-attention module in the coordinate decoder.

[0071] The cross-attention module is used to align and fuse semantic and visual information. Specifically, it involves aligning and fusing the hidden state corresponding to the visual positioning token. The query vector is constructed after linear mapping. Simultaneously, the enhanced image feature sequence output by the visual adapter is represented as... And mapped to key vectors and value vectors respectively: , .

[0072] Subsequently, the cross-attention module performs the following attention calculation process: ;in, This represents the visual-semantic fusion feature obtained after cross-attention fusion, which reflects the correspondence between the interface region of interest in the hidden state and the enhanced image features at the feature level.

[0073] Through the aforementioned cross-attention calculation, the semantic query vector can adaptively focus on the image feature region most relevant to the target interface element on a global scale, thereby suppressing the interference of irrelevant background regions on the localization result.

[0074] The output of the cross-attention module is a multi-location fused feature. To facilitate subsequent regression calculations, the fused feature can be subjected to feature aggregation processing, such as average pooling, weighted summation, or linear mapping, to obtain a single fused feature vector, which can be expressed as: ,in Represents the feature aggregation function, This is a fusion feature representation used for coordinate regression.

[0075] After completing the construction of the fusion features, the fusion features will be... The input is fed into the regression head. The regression head is a continuous numerical prediction structure, which may include one or more fully connected networks to map high-dimensional fused features into spatial coordinate parameters. Its calculation process can be expressed as follows: ,in This represents the regression mapping function, and the four consecutive floating-point numbers output correspond to the normalized top-left and bottom-right coordinates of the bounding box of the target interface element in the graphical user interface.

[0076] In this way, the coordinate decoder does not need to discretize the spatial location information into text symbols or coordinate strings, but directly outputs the bounding box parameters in the form of continuous numerical values, thereby avoiding the problems of quantization error, randomness of order and cumulative deviation introduced in the text generation process.

[0077] By using the coordinate decoder structure consisting of the cross-attention module and the regression head, the semantic understanding and spatial localization are effectively decoupled and efficiently connected while maintaining the end-to-end inference process of the multimodal large language model. This allows the present invention to simultaneously meet the requirements of high accuracy and low latency in graphical user interface localization tasks.

[0078] As an example, the cross-attention module is a multi-scale hierarchical structure, including at least a first cross-attention submodule and a second cross-attention submodule.

[0079] The first cross-attention submodule uses the hidden state corresponding to the visual positioning token as the query vector and the low-resolution image feature sequence as the key vector and value vector to perform cross-attention calculation, which is used to determine the candidate region of the target element in the graphical user interface.

[0080] The second cross-attention submodule uses the query vector shared with or linearly transformed by the first cross-attention submodule as the query vector, and performs cross-attention calculation only on the high-resolution image feature subset corresponding to the candidate region, in order to finely locate the target element and generate fusion features for bounding box regression.

[0081] In this implementation, the cross-attention module is designed as a multi-scale hierarchical structure to adapt to the spatial characteristics of interface elements in the graphical user interface, which presents a hierarchical nesting between the overall layout and local details, thereby improving the accuracy and efficiency of interface element positioning.

[0082] Specifically, the cross-attention module includes at least a first cross-attention submodule and a second cross-attention submodule, which are executed sequentially from coarse to fine. The processing of the first cross-attention submodule is as follows: First, based on the output of the visual adapter, a low-resolution image feature sequence describing the overall structure of the graphical user interface is constructed. This low-resolution image feature sequence can be obtained from the enhanced image feature sequence through downsampling, region pooling, or feature aggregation, and can be represented as: ;in, This represents the image feature vector corresponding to the i-th low-resolution interface region.

[0083] Subsequently, the hidden state corresponding to the visual positioning token is extracted. This is then used as a query vector and input into the first cross-attention submodule. Low-resolution image feature sequence These are mapped to key vectors and value vectors, respectively. The first cross-attention submodule performs the following attention calculation process: ;in, This represents the output of the first cross-attention submodule, which is the attention fusion feature obtained by fusing visual positioning token semantics and interface image features at a low resolution scale. It is used to characterize the candidate region response of the target interface element in the overall interface layout. It represents the result of a linear mapping from the hidden state corresponding to the visual positioning token in the current decoding step. It is used to express the semantic query requirements of the current positioning task and reflects the semantic information of the target interface elements that the user command is interested in. This indicates that the feature sequence of the low-resolution image is obtained through linear mapping. It is used to characterize the spatial and structural features of each region of the graphical user interface at the overall scale, and serves as a reference for relevance matching with the query vector. This means that it is also obtained by linear mapping of the feature sequence of low-resolution image, and it corresponds one-to-one with the key vector. It is used to carry specific interface region feature information after the attention weight is determined. For feature dimensions.

[0084] Through the first cross-attention calculation described above, the attention response distribution of the target element in the low-resolution interface space is obtained. Based on this attention response distribution, one or more candidate regions that are most relevant to the semantics of the target element are determined.

[0085] After obtaining the candidate regions, a subset of high-resolution image features corresponding to the candidate regions in spatial location is selected from the enhanced image feature sequence output by the visual adapter. This subset can be represented as follows: ,in It only includes image features located within the candidate region, used to describe fine-grained information such as the edge contours, rectangular boundaries, and text layout of target interface elements.

[0086] The processing procedure of the second cross-attention submodule is as follows: The second cross-attention submodule uses the query semantics shared with the first cross-attention submodule as input. Specifically, the query vector can directly reuse the query vector from the first cross-attention submodule. Alternatively, a new query vector can be obtained through linear transformation. ,in It is a linear transformation matrix used to adapt the query semantics.

[0087] Subsequently, a subset of high-resolution image features The mapping is performed as key vectors and value vectors, and a second cross-attention computation is performed only within the candidate region. The process can be represented as follows: ;in, This represents the output of the second cross-attention submodule, which is the attention fusion feature obtained by fusing the semantics of the visual positioning token with the high-resolution image features corresponding to the candidate region at a high-resolution scale. It is used to characterize the fine spatial location and structural information of the target interface elements. This indicates that the query vector used by the first cross-attention submodule is shared or obtained through linear transformation. Its source is still the hidden state of the visual localization token in the current decoding step, which is used to maintain the consistency of semantic targets during multi-scale localization. This represents a subset of high-resolution image features corresponding to the candidate regions determined by the first cross-attention submodule in spatial location, obtained by linear mapping, and is used to characterize the visual structural features of each fine-grained interface region within the candidate region. It represents a subset of high-resolution image features obtained through linear mapping, corresponding one-to-one with the key vector, and is used to carry fine feature information such as edge contours, rectangular boundaries, and text layout of target interface elements after the attention weights are determined.

[0088] The second cross-attention submodule obtains a high-precision fusion feature representation for the target interface element. This fusion feature contains both semantic constraint information from the visual positioning token and spatial detail information from the high-resolution image features.

[0089] After completing the second cross-attention fusion, the high-precision fused features are input into the regression head, which outputs the normalized bounding box coordinates of the target interface elements. Since the regression head performs regression calculations only based on candidate region features filtered through multi-scale hierarchical cross-attention, it effectively reduces computational complexity and improves overall inference efficiency while ensuring positioning accuracy.

[0090] Through the above-described multi-scale hierarchical cross-attention processing, this embodiment can quickly lock the candidate region of the target element at the overall interface layout level and achieve precise boundary positioning at the local detail level, avoiding the need to perform high-resolution cross-attention calculations globally, thereby significantly improving the real-time performance and accuracy of graphical user interface positioning tasks.

[0091] As an example, when the cross-attention module further includes at least one third cross-attention submodule, it also includes a coordination submodule.

[0092] The coordination submodule is configured to receive the output of at least one preceding cross-attention submodule, summarize or transform the output, and generate candidate region constraint information or attention modulation information for subsequent cross-attention submodules.

[0093] The candidate region constraint information or attention modulation information is passed as one of the input parameters to the subsequent cross-attention submodule, which is used to limit or modulate the selection range of key vectors and value vectors or the attention weight calculation process in the subsequent cross-attention submodule.

[0094] In this embodiment, when the cross-attention module includes a first cross-attention sub-module, a second cross-attention sub-module, and at least one third cross-attention sub-module, a coordination sub-module is set in the cross-attention module to achieve information coordination among the multi-level cross-attention sub-modules.

[0095] The coordination submodule is a functional module independent of each cross-attention submodule. Please refer to [link / reference]. Figure 3 Structurally, it includes at least: a feature aggregation unit, a feature transformation unit, and a constraint generation unit. These units are connected in sequence and are used to process the output of the preceding cross-attention submodule.

[0096] The feature transformation unit is connected to the feature convergence unit and is used to transform or scale the converged intermediate feature representation to adapt it to the input requirements of subsequent cross-attention submodules. The feature transformation unit may include a linear mapping layer or a normalization layer, used to map the features output by preceding cross-attention submodules to a representation with the same dimensions as the key and value vectors in subsequent cross-attention submodules, or to a weight form used for modulating attention computation. It can be understood that, through the feature transformation unit, the output information from different cross-attention submodules maintains consistency in the representation space.

[0097] The constraint generation unit is connected to the feature transformation unit and is used to generate candidate region constraint information or attention modulation information based on the transformed features.

[0098] The candidate region constraint information is used to indicate the range of image features allowed to participate in attention calculation in subsequent cross-attention submodules; the attention modulation information is used to impose constraints or biases on the calculation process of subsequent cross-attention submodules during the attention weight calculation stage. It is understood that the candidate region constraint information or attention modulation information output by the constraint generation unit is output as independent control parameters.

[0099] In this embodiment, the output of the coordination submodule is connected to the input of the subsequent cross-attention submodule. Candidate region constraint information or attention modulation information is passed to the subsequent cross-attention submodule as one of the input parameters. When performing cross-attention calculation, the subsequent cross-attention submodule restricts the selection range of key vectors and value vectors based on the candidate region constraint information, or participates in the calculation of attention weights based on the attention modulation information.

[0100] For example: Let the output feature of the i-th preceding cross-attention submodule be represented as .in, This indicates the number of image features that the cross-attention submodule focuses on. Indicates the feature dimension.

[0101] When multiple preceding cross-attention submodules exist, the feature aggregation unit aggregates the output features from different preceding cross-attention submodules. As an example, aggregation can be performed using concatenation or weighted summation. For instance, in a weighted summation implementation, the aggregation process can be represented as: ;in, The number of preceding cross-attention submodules. For the corresponding weight coefficients, satisfying This is used to reflect the importance of the output results of different cross-attention submodules.

[0102] Through the above processing by the feature aggregation unit, a unified intermediate feature representation is obtained. This is used to characterize the interface region information that multi-level cross-attention focuses on in the preceding stage.

[0103] The feature transformation unit is used to represent the intermediate features. Perform dimensional alignment or form transformation. As an example, the feature transformation unit can use a linear mapping approach, and its processing can be represented as follows: ,in For a trainable mapping matrix, For bias terms, The feature dimension is consistent with the key vector and value vector in the subsequent cross-attention submodule.

[0104] Through the feature transformation of the feature transformation unit, the outputs from different cross-attention submodules have a unified scale and representation in the feature space.

[0105] The constraint generation unit is based on the transformed features This generates candidate region constraint information or attention modulation information. In one implementation, the candidate region constraint information can be represented as a binary or continuous mask: ,in For mapping parameters, The mask generated by the activation function. Used to indicate the locations of image features that are allowed to participate in attention calculations in subsequent cross-attention submodules.

[0106] Attention modulation information can be represented as attention bias terms: ,in These are trainable parameters used to introduce modulation terms in subsequent cross-attention weight calculations.

[0107] In the subsequent cross-attention submodule, candidate region constraint information or attention modulation information is used as one of the input parameters in the cross-attention calculation. Taking the attention weight calculation as an example, the calculation process can be expressed as follows: Or, when introducing candidate region constraint information, it can be represented as: Among them, This serves as the query vector for the subsequent cross-attention submodule. and They are the key vector and the value vector, respectively. This indicates element-wise multiplication.

[0108] In this implementation, when the cross-attention module includes three or more cross-attention sub-modules and introduces a coordination sub-module, a clear structured information transmission and constraint mechanism can be established between the multi-level cross-attention sub-modules. This ensures that the output results of the preceding cross-attention sub-modules are no longer isolated from each other, but are uniformly converged, transformed, and converted into candidate region information or attention modulation parameters that have a constraint effect on subsequent cross-attention sub-modules through the coordination sub-module. This avoids the problems of attention region drift and redundant calculation caused by the simple stacking of multi-level cross-attention. While ensuring the hierarchical expansion of the localization process, it improves the certainty and consistency of feature fusion in the key interface region by the subsequent cross-attention sub-modules, which is conducive to the stable output of high-quality bounding box regression features and improves the reliability of the interface element localization results and the controllability of the overall reasoning process.

[0109] Please see Figure 4This invention also provides a GUI positioning system 200 based on a route prediction framework. The system includes an input acquisition and encoding module 201, which is used to acquire images of the graphical user interface and user command text, and preprocess and encode them to obtain image feature sequences and text feature sequences respectively, and concatenate the image feature sequences and text feature sequences to obtain a fused feature sequence.

[0110] The multimodal inference module 202 is used to input the fused feature sequence into the multimodal large language model and obtain the hidden state output by the multimodal large language model in the current decoding step.

[0111] The token routing module 203 is used to determine the type of token output in the current decoding step based on the hidden state, including visual positioning token, rejection response token and ordinary text token.

[0112] The positioning decoding module 204 is used to trigger the coordinate decoder if the token type is a visual positioning token, to fuse and decode the hidden state corresponding to the visual positioning token with the image features enhanced by the visual adapter, and output continuous spatial coordinate values ​​as the bounding box of the target element.

[0113] If the token type is a rejection response token, an empty location result is returned, and the location request is rejected.

[0114] If the token type is a plain text token, then the multimodal large language model continues to autoregressively generate semantic description text.

[0115] This invention also provides a medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the preceding claims.

[0116] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the preceding claims.

[0117] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A GUI positioning method based on a routing prediction framework, characterized in that, The method includes the following steps: Step S1: Obtain the image of the graphical user interface and the user command text, and perform preprocessing and encoding to obtain the image feature sequence and the text feature sequence respectively. Concatenate the image feature sequence and the text feature sequence to obtain the fused feature sequence. Step S2: Input the fused feature sequence into the multimodal large language model to obtain the hidden state output by the multimodal large language model in the current decoding step; Step S3: Based on the hidden state, determine the token type output in the current decoding step through the token routing unit, including visual positioning token, rejection response token, and ordinary text token; Step S4: If the token type is a visual positioning token, the coordinate decoder is triggered to fuse and decode the hidden state corresponding to the visual positioning token with the image features enhanced by the visual adapter, and output continuous spatial coordinate values ​​as the bounding box of the target element; if the token type is a rejection response token, an empty positioning result is returned, and the positioning request is rejected; if the token type is a plain text token, the multimodal large language model continues to autoregressively generate semantic description text. The token routing unit is implemented by reusing the output projection layer of the multimodal large language model; wherein, the original vocabulary of the output projection layer is expanded, and two new vocabulary entries corresponding to the visual positioning token and the rejection response token are added; The visual adapter is a lightweight neural network module. Its input is the original image feature sequence output by the visual encoder of the multimodal large language model. It is processed through a structure containing at least one bidirectional self-attention layer and outputs enhanced image features for the edges, rectangular regions and text layout of target elements in the graphical user interface. The coordinate decoder includes a cross-attention module and a regression head; the cross-attention module uses the hidden state corresponding to the visual positioning token as the query vector and the enhanced image features as the key vector and value vector for feature fusion; the regression head is used to map the fused features into four consecutive floating-point numbers, which directly constitute normalized bounding box coordinates. The cross-attention module is a multi-scale hierarchical structure, including at least a first cross-attention submodule and a second cross-attention submodule. The first cross-attention submodule uses the hidden state corresponding to the visual positioning token as the query vector and a low-resolution image feature sequence as the key and value vectors for cross-attention calculation, used to determine candidate regions of target elements in the graphical user interface. The second cross-attention submodule uses a query vector shared with or linearly transformed by the first cross-attention submodule as the query vector, and performs cross-attention calculation only on a subset of high-resolution image features corresponding to the candidate regions, used for fine-tuning the target element and generating fused features for bounding box regression.

2. The GUI localization method based on a route prediction framework according to claim 1, characterized in that: When the cross-attention module further includes at least one third cross-attention submodule, it also includes a coordination submodule; wherein, the coordination submodule is configured to receive the output results of at least one preceding cross-attention submodule, summarize or transform the output results, and generate candidate region constraint information or attention modulation information for subsequent cross-attention submodules; the candidate region constraint information or attention modulation information is passed to the subsequent cross-attention submodule as one of the input parameters, and is used to limit or modulate the selection range of key vectors and value vectors or the attention weight calculation process in the subsequent cross-attention submodule.

3. A GUI positioning system based on a routing prediction framework, the system based on the method of claim 1 or 2, characterized in that: The system includes: The input acquisition and encoding module is used to acquire images and user command text from the graphical user interface, and preprocess and encode them to obtain image feature sequences and text feature sequences respectively. The image feature sequences and text feature sequences are then concatenated to obtain a fused feature sequence. The multimodal reasoning module is used to input the fused feature sequence into the multimodal large language model and obtain the hidden state output by the multimodal large language model in the current decoding step; The token routing module is used to determine the type of token output in the current decoding step based on the hidden state, including visual positioning tokens, rejection response tokens, and plain text tokens. The localization decoding module is used to trigger the coordinate decoder if the token type is a visual localization token, fuse and decode the hidden state corresponding to the visual localization token with the image features enhanced by the visual adapter, and output continuous spatial coordinate values ​​as the bounding box of the target element; if the token type is a rejection response token, an empty localization result is returned, and the localization request is rejected; if the token type is a plain text token, the multimodal large language model continues to autoregressively generate semantic description text.

4. A medium having stored thereon a computer program, characterized by: When the computer program is executed by a processor, it implements the method as described in claim 1 or 2.

Images