Visual language model positioning capability evaluation method and device, computer equipment and medium
By generating saliency heatmaps to evaluate the localization ability of visual language models, this approach solves the problem of difficulty in evaluating the localization ability of models in graphical user interfaces in existing technologies, and improves the accuracy of model operation in graphical user interfaces.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-17
- Publication Date
- 2026-03-31
AI Technical Summary
Existing technologies struggle to effectively assess the positioning capabilities of visual language models in graphical user interfaces, resulting in insufficient operational accuracy.
By extracting the cross-modal attention tensor of the cross-modal cross-attention network layer of the visual language model, a saliency heatmap is generated, and the hit rate of the model in target element localization in the graphical user interface is evaluated based on the heatmap.
An internal evaluation of the potential localization capability of the visual language model was achieved, improving the accuracy of the model in graphical user interface operations.
Smart Images
Figure CN121764579A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, specifically to a method, apparatus, computer device, and storage medium for evaluating the localization capability of a visual language model. Background Technology
[0002] A graphical user interface (GUI) is an interface design that allows users to interact with electronic devices, such as computers, mobile phones, and tablets, through graphical elements. It uses visual icons, buttons, windows, menus, etc., to replace complex code or instructions, making operation more intuitive.
[0003] With the significant breakthroughs achieved by large-scale visual language models (VLMs) in tasks such as image captioning, visual question answering, and text-to-image processing, using VLMs for GUI operations has become a widely applied technique. The core of using VLMs for GUI operations is localization, that is, determining the position of target elements on the GUI page based on the VLM. The accuracy of localization directly affects the accuracy of the operation.
[0004] Therefore, a method for evaluating the localization capability of natural language models for GUI positioning has become an urgent problem to be solved. Summary of the Invention
[0005] In view of this, this application provides a method, apparatus, computer device, and storage medium for evaluating the localization capability of a visual language model.
[0006] Specifically, this application is implemented through the following technical solution: In a first aspect, embodiments of this disclosure provide a method for evaluating the localization capability of a visual language model, the method comprising: The sample data is input into the visual language model to obtain the cross-modal attention tensor output by the cross-modal cross-attention network layer in the visual language model; the sample data includes: images of the graphical user interface and corresponding natural language instructions; the cross-modal attention tensor is used to characterize the distribution of attention to text units corresponding to the natural language instructions and image units corresponding to the images of the graphical user interface. Based on the cross-modal attention tensor, a saliency heatmap of the cross-attention network layer is generated; the saliency heatmap is used to describe the overall visual attention distribution to the image of the graphical user interface. Based on the saliency heatmap, a location hit determination process is performed to obtain the hit rate of the cross-attention network layer in locating target elements in the graphical user interface during the cross-modal cross-attention processing based on the sample data; the hit rate is used to describe the localization capability of the visual language model.
[0007] Optionally, there may be multiple cross-attention network layers; The step of generating a saliency heatmap of the cross-modal attention network layer based on the cross-modal attention tensor includes: The cross-modal attention tensor is split to obtain attention weight matrices corresponding to the multiple cross-attention network layers; Attention weight matrices corresponding to each of the cross-attention network layers are aggregated to obtain aggregated attention weight matrices; and saliency heatmaps corresponding to each of the cross-attention network layers are generated based on the aggregated attention weight matrices.
[0008] Optionally, the attention weight matrix corresponding to each of the cross-attention network layers includes: the attention weight matrix from the text unit to all image units in each of the plurality of attention heads; The step of performing attention aggregation processing on the attention weight matrix corresponding to each of the cross-attention network layers to obtain the aggregated attention weight matrix includes: The attention weight matrices from text units to all image units corresponding to multiple attention heads are weighted to obtain the aggregated attention weight matrix.
[0009] Optionally, generating a saliency heatmap for each of the cross-attention network layers based on the aggregated attention weight matrix includes: The aggregated attention weight matrix is restructured to obtain a two-dimensional spatial structure attention weight matrix; Based on the size of the image corresponding to the graphical user interface, the attention weight matrix of the two-dimensional spatial structure is interpolated to obtain the saliency heatmap corresponding to each of the cross-attention network layers.
[0010] Optionally, the location hit determination process based on the saliency heatmap, to obtain the hit rate of the cross-attention network layer in locating target elements in the graphical user interface during cross-modal cross-attention processing based on the sample data, includes: Based on the saliency heatmaps corresponding to each cross-attention network layer and the actual location information of the target element corresponding to the natural language command in the image of the graphical user interface, the localization hit information of each cross-attention network layer when performing cross-attention processing based on the sample data is determined; the localization hit information includes: hit or miss. Based on the localization hit information corresponding to the multiple cross-attention network layers, the hit information of each sample data is determined, and based on the hit information corresponding to the multiple sample data, the hit rate is determined.
[0011] Optionally, determining the localization hit information of each cross-attention network layer when performing cross-attention processing based on the sample data, according to the saliency heatmaps corresponding to each cross-attention network layer and the actual position information of the target element corresponding to the natural language command in the image of the graphical user interface, includes: Extract the coordinates of the point with the maximum response from the saliency heatmap corresponding to each of the cross-attention network layers; Based on the consistency between the position information of the target element in the image of the graphical user interface and the coordinate value of the extracted maximum response point, the hit information of each cross-attention network layer when performing cross-attention processing based on the sample data is determined.
[0012] Optionally, the location information of the target element in the image of the graphical user interface includes: a binary mask image; and the pixel values of each pixel in the binary mask image, used to characterize whether the pixel belongs to the target element. The step of determining the hit information corresponding to each of the cross-attention network layers based on the consistency between the position information of the target element in the image of the graphical user interface and the coordinate value of the extracted maximum response point includes: Based on the extracted coordinates of the maximum response point, the pixel values of the pixels corresponding to the coordinates are read from the binary mask image. Based on the coordinate values of the pixels corresponding to the read coordinate values, the hit information corresponding to each of the cross-attention network layers is determined.
[0013] Optionally, determining the hit information of each sample data based on the localization hit information corresponding to the multiple cross-attention network layers includes: When multiple cross-attention network layers perform cross-attention processing based on the same sample data, determine whether the localization hit information of at least one of the cross-attention network layers is a hit; if so, determine that the hit information of the sample data is a hit. If multiple cross-attention network layers fail to find the localization hit information when performing cross-attention processing on the general sample data, then the hit information of the sample data is determined to be a miss.
[0014] Secondly, embodiments of this disclosure also provide an evaluation device for the localization capability of a visual language model, the device comprising: The processing module is used to input sample data into the visual language model to obtain the cross-modal attention tensor output by the cross-modal cross-attention network layer in the visual language model; the sample data includes: images of the graphical user interface and corresponding natural language instructions; the cross-modal attention tensor is used to characterize the distribution of attention to text units corresponding to the natural language instructions and image units corresponding to the images of the graphical user interface. A generation module is used to generate a saliency heatmap of the cross-modal attention network layer based on the cross-modal attention tensor; the saliency heatmap is used to describe the overall visual attention distribution of the image of the graphical user interface; An evaluation module is used to determine the location hit rate based on the saliency heatmap, and to obtain the hit rate of the cross-attention network layer in locating target elements in the graphical user interface during the cross-modal cross-attention processing based on the sample data; the hit rate is used to describe the localization capability of the visual language model.
[0015] Optionally, there may be multiple cross-attention network layers; The generation module, when generating the saliency heatmap of the cross-attention network layer based on the cross-modal attention tensor, is used to: The cross-modal attention tensor is split to obtain attention weight matrices corresponding to the multiple cross-attention network layers; Attention weight matrices corresponding to each of the cross-attention network layers are aggregated to obtain aggregated attention weight matrices; and saliency heatmaps corresponding to each of the cross-attention network layers are generated based on the aggregated attention weight matrices.
[0016] Optionally, the attention weight matrix corresponding to each of the cross-attention network layers includes: the attention weight matrix from the text unit to all image units in each of the plurality of attention heads; The generation module, when performing attention aggregation processing on the attention weight matrix corresponding to each of the cross-attention network layers to obtain the aggregated attention weight matrix, is used for: The attention weight matrices from text units to all image units corresponding to multiple attention heads are weighted to obtain the aggregated attention weight matrix.
[0017] Optionally, the generation module, when generating a saliency heatmap corresponding to each of the cross-attention network layers based on the aggregated attention weight matrix, is used to: The aggregated attention weight matrix is restructured to obtain a two-dimensional spatial structure attention weight matrix; Based on the size of the image corresponding to the graphical user interface, the attention weight matrix of the two-dimensional spatial structure is interpolated to obtain the saliency heatmap corresponding to each of the cross-attention network layers.
[0018] Optionally, the evaluation module, when determining the hit rate of the cross-attention network layer in locating target elements in the graphical user interface during cross-modal cross-attention processing based on the saliency heatmap, is used to: Based on the saliency heatmaps corresponding to each cross-attention network layer and the actual location information of the target element corresponding to the natural language command in the image of the graphical user interface, the localization hit information of each cross-attention network layer when performing cross-attention processing based on the sample data is determined; the localization hit information includes: hit or miss. Based on the localization hit information corresponding to the multiple cross-attention network layers, the hit information of each sample data is determined, and based on the hit information corresponding to the multiple sample data, the hit rate is determined.
[0019] Optionally, the evaluation module, when determining the localization hit information of each cross-attention network layer when performing cross-attention processing based on the sample data, according to the saliency heatmaps corresponding to each cross-attention network layer and the real location information of the target element corresponding to the natural language instruction in the image of the graphical user interface, is used to: Extract the coordinates of the point with the maximum response from the saliency heatmap corresponding to each of the cross-attention network layers; Based on the consistency between the position information of the target element in the image of the graphical user interface and the coordinate value of the extracted maximum response point, the hit information of each cross-attention network layer when performing cross-attention processing based on the sample data is determined.
[0020] Optionally, the location information of the target element in the image of the graphical user interface includes: a binary mask image; and the pixel values of each pixel in the binary mask image, used to characterize whether the pixel belongs to the target element. The evaluation module, when determining the hit information corresponding to each of the cross-attention network layers based on the consistency between the position information of the target element in the image of the graphical user interface and the coordinate value of the extracted maximum response point, is used to: Based on the extracted coordinates of the maximum response point, the pixel values of the pixels corresponding to the coordinates are read from the binary mask image. Based on the coordinate values of the pixels corresponding to the read coordinate values, the hit information corresponding to each of the cross-attention network layers is determined.
[0021] Optionally, the evaluation module, when determining the hit information of each sample data based on the localization hit information corresponding to the multiple cross-attention network layers, is used to: When multiple cross-attention network layers perform cross-attention processing based on the same sample data, determine whether the localization hit information of at least one of the cross-attention network layers is a hit; if so, determine that the hit information of the sample data is a hit. If multiple cross-attention network layers fail to find the localization hit information when performing cross-attention processing on the general sample data, then the hit information of the sample data is determined to be a miss.
[0022] Thirdly, an optional implementation of this disclosure also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the first aspect above, or any possible implementation of the first aspect.
[0023] Fourthly, an optional implementation of this disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the first aspect or any possible implementation of the first aspect.
[0024] Fifthly, an optional implementation of this disclosure also provides a computer program product, the computer program product carrying program code, the instructions included in the program code being able to execute the steps of the first aspect above, or any possible implementation of the first aspect.
[0025] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure.
[0026] The visual language model localization capability evaluation method provided in this disclosure extracts the cross-modal attention tensor output by the cross-modal cross-attention network layer within the visual language model, and generates a saliency heatmap of the cross-attention network layer based on the cross-modal attention tensor. This saliency heatmap is used to describe the overall visual attention distribution of the image of the graphical user interface. Then, based on the saliency heatmap, the hit rate of the cross-attention network layer in locating target elements in the graphical user interface during the cross-modal cross-attention processing is determined, so as to evaluate the potential localization capability of the model using internal model signals.
[0027] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description
[0028] Figure 1 A flowchart is shown showing a method for evaluating the localization capability of a visual language model provided in some embodiments of this disclosure; Figure 2 The present disclosure shows a schematic diagram of the structure of a computer device provided in some embodiments; Figure 3 A schematic diagram of the structure of an evaluation apparatus for the localization capability of a visual language model provided in some embodiments of this disclosure is shown. Detailed Implementation
[0029] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0030] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.
[0031] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."
[0032] In related technologies, with the widespread application of Vision-Language Models (VLMs) in tasks such as image understanding, graph analysis, and graphical user interface (GUI) operation, the model's spatial grounding ability has become one of the key factors affecting performance.
[0033] In GUI tasks, visual language models need to understand natural language instructions, such as "click the download button in the upper right corner," and accurately locate target controls within complex graphical user interfaces. However, most current multimodal models can only generate language output or global responses, lacking explicit alignment mechanisms for visual regions, making it difficult to determine whether the model truly "sees" the corresponding interface elements. Existing methods for evaluating the localization capabilities of visual language models largely rely on task execution results, such as click success rate or detection accuracy, which only measure the model's external performance and cannot reveal whether the model possesses potential spatial understanding capabilities. Therefore, an evaluation method capable of assessing the model's inherent potential localization capabilities has become a pressing issue.
[0034] To address the aforementioned issues, this disclosure provides a method for evaluating the localization capability of a visual language model. This method extracts the cross-modal attention tensor output by the cross-modal cross-attention network layer within the visual language model and generates a saliency heatmap of the cross-attention network layer based on this tensor. This saliency heatmap describes the overall visual attention distribution to the image of the graphical user interface. Then, based on the saliency heatmap, the hit rate of the cross-attention network layer in locating target elements in the graphical user interface during cross-modal cross-attention processing is determined, thereby enabling the evaluation of the model's potential localization capability using internal model signals.
[0035] The shortcomings of the above solutions are the result of the inventor's practical experience and careful research. Therefore, the discovery process of the above problems and the solutions proposed in this disclosure below should be considered as the inventor's contribution to this disclosure.
[0036] To facilitate understanding of the technical solutions disclosed herein, the technical terms used in the embodiments of this disclosure will first be explained: Visual language models (VLMs) are a class of multimodal artificial intelligence models that integrate visual information processing with natural language understanding / generation capabilities. Their core objective is to break down the modal barriers between visual data (images, videos, etc.) and natural language data (text, speech, etc.), achieving cross-modal semantic alignment, interaction, and reasoning. The essence of VLMs is to establish a mapping relationship between visual features and linguistic features, deeply fusing information from both modalities through model architecture, ultimately possessing three core capabilities: Cross-modal understanding: Simultaneously analyzing the semantic relationships between visual and linguistic information (such as determining whether "there is a red cup in the picture" or "whether the text description matches the image"). Cross-modal generation: generating language content (such as image descriptions and video summaries) based on visual input, or editing visual content (such as modifying image elements based on text) based on language instructions; Cross-modal reasoning: Completing complex logical tasks based on the combination of vision and language (such as "answering mathematical questions based on chart data" or "analyzing the causal relationships of events in a video").
[0037] The core of using VLM for GUI operations is to rely on VLM's cross-modal understanding capabilities, combined with dedicated tools or customized training schemes, to enable the model to map natural language commands to visual elements of the GUI interface, locate the position of the target visual element in the GUI, and then use the location results to complete operations such as clicking and input.
[0038] To facilitate understanding of this embodiment, a detailed description of the visual language model localization capability evaluation method disclosed in this disclosure is provided first. The execution entity of the visual language model localization capability evaluation method provided in this disclosure is generally a computer device with a certain computing power. This computer device may include, for example, a terminal device, a server, or other processing devices. The terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, an in-vehicle device, a wearable device, etc. In some possible implementations, the visual language model localization capability evaluation method can be implemented by a processor calling computer-readable instructions stored in memory.
[0039] It should be noted that the visual language model localization capability evaluation method provided in this disclosure can be used not only to evaluate the localization capability of graphical user interfaces, but also to evaluate the localization capability of other types of images. Specific embodiments of this disclosure are not limited.
[0040] The following describes the method for evaluating the localization capability of a visual language model provided in the embodiments of this disclosure.
[0041] See Figure 1 The diagram shows a flowchart of a visual language model localization capability evaluation method provided in this embodiment of the present disclosure. The method includes steps S101 to S103, wherein: S101: Input sample data into the visual language model to obtain the cross-modal attention tensor output by the cross-modal cross-attention network layer in the visual language model; the sample data includes: images of the graphical user interface and corresponding natural language instructions; the cross-modal attention tensor is used to characterize the distribution of attention to text units corresponding to the natural language instructions and image units corresponding to the images of the graphical user interface. S102: Generate a saliency heatmap of the cross-modal attention network layer based on the cross-modal attention tensor; the saliency heatmap is used to describe the overall visual attention distribution of the image of the graphical user interface; S103: Based on the saliency heatmap, perform location hit determination processing to obtain the hit rate of the cross-attention network layer in locating target elements in the graphical user interface during the cross-modal cross-attention processing based on the sample data; the hit rate is used to describe the localization capability of the visual language model.
[0042] This embodiment extracts the cross-modal attention tensor output by the cross-modal cross-attention network layer within the visual language model. This cross-modal attention tensor characterizes the distribution of attention given by text units corresponding to the natural language instructions to image units corresponding to the images in the graphical user interface. Then, based on this cross-modal attention tensor, a saliency heatmap of the cross-attention network layer is generated. This saliency heatmap describes the overall visual attention distribution of text units to the images in the graphical user interface. Subsequently, based on the saliency heatmap, a location hit determination process is performed to obtain the hit rate of the cross-attention network layer in locating target elements in the graphical user interface during the cross-modal cross-attention processing. This allows for the evaluation of the model's potential localization capability using internal model signals.
[0043] The following provides a detailed explanation of S101 to S103.
[0044] Regarding the above S101: In practice, the sample data typically consists of multiple sets; each set includes: an image of the graphical user interface and a corresponding natural language instruction. Additionally, it may include label information indicating the location of the target element corresponding to the natural language instruction within the image of the graphical user interface. This label information describes the actual location of the target element corresponding to the natural language instruction within the image of the graphical user interface.
[0045] The images and corresponding natural language instructions in the graphical user interface can be, for example, pre-acquired sample data used to evaluate the localization capabilities of the visual language model.
[0046] Furthermore, the evaluation of visual language models can also occur during the model application phase. That is, images of the graphical user interface (GUI) and natural language commands acquired during model application can be used as sample data to evaluate the localization capability of the visual language model. Specifically, based on the model execution results (whether localization was successful or not), label information corresponding to the application phase can be generated. For example, if the system using the visual language model for GUI control successfully executes natural language commands, the localization result of the target element output by the visual language model is considered accurate, and corresponding label information is generated based on the localization result of the target element output by the visual language model.
[0047] During the application phase of the model, the image of the graphical user interface (GUI) can be acquired, for example, after the computer device's interface transitions to the GUI. The GUI image can be automatically acquired each time an interface transition event occurs. Alternatively, the GUI image can be acquired only after a specific event is triggered. For example, some GUIs include controls for user input; if the user enters information into these controls and then needs to perform other operations, such as deleting the entered information, the GUI image in its current state can be acquired in real-time when this specific event is triggered by a command.
[0048] Natural language commands used to interact with graphical user interfaces can be in text or speech form. For natural language commands in speech form, a corresponding speech recognition model can be used to convert the speech commands into text.
[0049] When evaluating the localization capability of a visual language model, images from a graphical user interface and their corresponding natural language commands are input as sample data into the visual language model to be evaluated. The visual language model then processes the user interface images and natural language commands. During this processing, the visual language model internally generates a cross-modal attention tensor.
[0050] Specifically, the core of the visual language model's ability to achieve cross-modal processing lies in its cross-attention mechanism. After inputting images and text into the visual language model, it can convert the images and text into a sequence of units (tokens) of the same dimension.
[0051] On the visual side, images are decomposed into discrete feature units, i.e., image tokens. On the text side, sentences are decomposed into semantic units, i.e., text tokens. Both are mapped to the same feature space, providing a universal carrier for cross-modal interaction. Then, a cross-modal attention mechanism is used to achieve bidirectional alignment between the text and visual sides, and cross-attention is used to realize semantic association between modalities. From text to vision, text tokens actively "query" corresponding features in the vision; from vision to text, image tokens actively "match" corresponding semantics in the text.
[0052] Attention mechanisms satisfy the following formula (1): (1) (Query) represents the "query" of the current modality, such as a text token; (Key) represents the "key" of the target modality, such as the feature identifier of a visual token; (Value) represents the "value" of the target modality, such as the specific features of a visual token; This indicates a constant, used to avoid excessively large inner product values due to excessively high dimensions; This represents the activation function.
[0053] Cross-attention between text and vision: The text token is used as Q and the visual token is used as K / V for attention processing.
[0054] Visual → Text Cross-Attention: Visual tokens are used as Q and text tokens are used as K / V for attention processing.
[0055] Visual language models typically include at least one cross-modal cross-attention network layer. Each cross-modal cross-attention network layer can use features from one modality as a "queryer" to determine relevant information from features from another modality, ultimately achieving the final association between the two modalities.
[0056] In practical implementation, the cross-attention processing mechanism first splits the original natural language instructions and images into h groups through different linear projections, where h represents the number of attention heads. Different attention heads focus on different features. Each attention head calculates its attention independently, and finally, the outputs of all attention heads are integrated and fused through a linear layer to obtain more comprehensive features than a single attention head.
[0057] After breaking down the original natural language instructions and images, the resulting data consists of text units (i.e., text tokens) and image units (i.e., image tokens).
[0058] Then, using the above formula (1), the attention weights between each text token and each image token are calculated.
[0059] Each cross-modal attention network layer outputs an attention weight matrix corresponding to that layer. The attention weight matrices formed by multiple cross-modal attention network layers constitute the cross-modal attention tensor.
[0060] Assume that the cross-modal attention network layer has Layer, number of attention heads is In a set of sample data, the total number of image tokens corresponding to images in the graphical user interface is The attention tensor output by the cross-modal attention network layer is then expressed as: .
[0061] Here, the query is a text token; Both the key and the value are image tokens.
[0062] For multiple sets of sample data, the cross-attention network layer outputs attention tensors corresponding to each set of sample data.
[0063] Regarding S102 above: When generating saliency heatmaps for cross-modal attention network layers based on the cross-modal attention tensor, the following approach can be used, for example: The cross-modal attention tensor is split to obtain attention weight matrices corresponding to the multiple cross-attention network layers; Attention weight matrices corresponding to each of the cross-attention network layers are aggregated to obtain aggregated attention weight matrices; and saliency heatmaps corresponding to each of the cross-attention network layers are generated based on the aggregated attention weight matrices.
[0064] In a specific implementation, when the cross-modal attention tensor is split according to the cross-attention network layer, the attention weight matrix corresponding to each cross-attention network layer is obtained.
[0065] The attention weight matrix corresponding to each cross-attention network layer includes the attention weight matrix from the text unit to all image units in multiple attention heads. It can be represented as: It indicates that it is aimed at the first In the nth cross-attention network layer, the nth In the attention head, the text token To all image tokens Attention weight matrix.
[0066] Then, the attention weight matrix corresponding to each of the cross-attention network layers is aggregated in the following manner to obtain the aggregated attention weight matrix: The attention weight matrices from text units to all image units corresponding to multiple attention heads are weighted to obtain the aggregated attention weight matrix.
[0067] For example, the attention weight matrices corresponding to all attention heads can be averaged to obtain an aggregated attention weight matrix. For example, it satisfies the following formula (2): (2) In addition, other methods can be used to determine the weight of each attention head, such as the consistency of the attention distribution represented by the attention weights corresponding to different attention heads. That is, for attention weights with higher consistency with the attention weights corresponding to multiple attention heads, their corresponding weights can be higher, while for attention weights with lower consistency, their corresponding weights can be lower.
[0068] Furthermore, since the purpose is to evaluate the localization capability of the visual language model, more stringent evaluation conditions can be set. That is, the weighting weight of attention weights with low consistency can be increased, and / or the weighting weight of attention weights with low consistency can be decreased in response. This allows for the weighting of the attention weight matrices corresponding to multiple attention heads to obtain an aggregated attention weight matrix.
[0069] When determining the consistency of different attention weights, one can calculate the average of the attention weight matrices corresponding to multiple attention heads, and use the difference between each attention weight matrix and this average as the consistency information for each attention weight matrix. The larger the difference, the lower the consistency; the smaller the difference, the higher the consistency.
[0070] In addition, there may be other ways to determine consistency, which are not limited in the embodiments disclosed herein.
[0071] After obtaining the aggregated attention weight matrix using the above method, the saliency heatmap corresponding to each of the cross-attention network layers can be generated using the following method: The aggregated attention weight matrix is restructured to obtain a two-dimensional spatial structure attention weight matrix; Based on the size of the image corresponding to the graphical user interface, the attention weight matrix of the two-dimensional spatial structure is interpolated to obtain the saliency heatmap corresponding to each of the cross-attention network layers.
[0072] Specifically, when interpolating the attention weight matrix of the two-dimensional spatial structure according to the size corresponding to the image of the graphical user interface, for example, the attention weight matrix of the two-dimensional spatial structure is upsampled with the size corresponding to the image of the graphical user interface as the target, so as to obtain a saliency heatmap with the same size as the size corresponding to the graphical user interface.
[0073] For example, the first Saliency heatmap of each cross-attention network layer For example, it satisfies the following formula (3): (3) in, This represents an upsampling operation in space, which is also a method of interpolation. This represents the structural reshaping of the aggregated attention weight matrix, used to transform the aggregated attention weight matrix into a two-dimensional spatial structure.
[0074] Using the above method, the first... Saliency heatmaps for each cross-attention network layer.
[0075] This saliency heatmap, for example, is used to characterize the first... The output of each cross-attention network layer is the overall visual attention distribution of the text token to the image of the graphical user interface. The pixel value of each first pixel in the saliency heatmap represents the attention the text token gives to the second pixel in the graphical user interface corresponding to that first pixel.
[0076] The higher the visual attention given to a pixel, the higher the probability that the visual language model considers that pixel to be the target element corresponding to the text token. Conversely, the lower the visual attention given to a pixel, the lower the probability that the visual language model considers that pixel to be the target element corresponding to the text token.
[0077] Regarding the above S103: After obtaining the saliency heatmaps of each cross-attention network layer, the following method can be used to perform location hit determination based on the saliency heatmaps, thereby obtaining the hit rate of the cross-attention network layer in locating target elements in the graphical user interface during cross-modal cross-attention processing based on the sample data: Based on the saliency heatmaps corresponding to each cross-attention network layer and the actual location information of the target element corresponding to the natural language command in the image of the graphical user interface, the localization hit information of each cross-attention network layer when performing cross-attention processing based on the sample data is determined; the localization hit information includes: hit or miss. Based on the localization hit information corresponding to the multiple cross-attention network layers, the hit information of each sample data is determined, and based on the hit information corresponding to the multiple sample data, the hit rate is determined.
[0078] In specific implementation, since the saliency heatmap corresponding to the cross-attention network is used to describe the overall visual attention distribution of the image of the graphical user interface, and the higher the visual attention of a certain pixel, the greater the probability that the pixel belongs to the target element, it can be determined whether the pixel with the highest visual attention can match the real position information of the target element in the graphical user interface based on the overall visual attention distribution of the image of the graphical user interface described by the saliency heatmap.
[0079] If a match is found, it means the cross-attention network layer has successfully located the target element. If no match is found, it means the cross-attention network layer has failed to locate the target element.
[0080] Specifically, when determining the localization hit information of each cross-attention network layer when performing cross-attention processing based on the sample data, based on the saliency heatmap corresponding to each cross-attention network layer and the actual position information of the target element corresponding to the natural language command in the image of the graphical user interface, the following method can be used, for example: Extract the coordinates of the point with the maximum response from the saliency heatmap corresponding to each of the cross-attention network layers; Based on the consistency between the position information of the target element in the image of the graphical user interface and the coordinate value of the extracted maximum response point, the hit information of each cross-attention network layer when performing cross-attention processing based on the sample data is determined.
[0081] For example, when extracting the coordinates of the maximum response point from the saliency heatmap corresponding to each cross-attention network layer, the coordinates of the maximum response point... Satisfy the following formula (4): (4) A heatmap indicating significance.
[0082] The actual location information of the target element in the graphical user interface corresponding to the natural language command, i.e., the label information, can be given in the form of coordinates or a bounding box. For example, the coordinate values of the target element in the graphical user interface or the coordinate values of the bounding box can be given directly.
[0083] Alternatively, a binary mask image can be used, where the pixel value of each pixel in the binary mask image is used to characterize whether the pixel belongs to the target element.
[0084] For example, when the real location information is given in the form of a binary mask image, the pixel value corresponding to the pixel corresponding to the maximum response point can be read from the binary mask image based on the coordinate value of the extracted maximum response point. Based on the read pixel value corresponding to the coordinate value, the hit information of the cross-attention network layer when performing cross-attention processing based on the sample data can be determined.
[0085] For example, in a binary mask image, if a pixel is 1, it means that the pixel belongs to the target element; if a pixel is 0, it means that the pixel does not belong to the target element.
[0086] After extracting the coordinates of the maximum response point, the pixel located at that position can be determined from the binary mask image based on the coordinates, and the pixel value of that pixel in the binary mask image can be read.
[0087] If the pixel value is 0, it means the hit information is a miss; if the pixel value is 1, it means the hit information is a hit.
[0088] For example, suppose the binary mask image is represented as: The actual location information of the target element is represented as the mask region in the binary mask image;
[0089] If the maximum response point falls within the masked area, it is considered a hit. This includes the hit information of any cross-attention network layer for a given sample data. Represented as: .
[0090] : indicates a hit function, where the function returns 1 if a hit occurs and 0 if a miss occurs.
[0091] Using the above method, we can obtain the hit information corresponding to each of the multiple cross-attention network layers when performing cross-attention processing based on the same sample data.
[0092] Then, based on the hit information corresponding to the sample data when multiple cross-attention network layers perform cross-attention processing on the same sample data, the hit information corresponding to the sample data is obtained.
[0093] Specifically, when determining the hit information of each sample data based on the localization hit information corresponding to the multiple cross-attention network layers, the following method can be used, for example: When multiple cross-attention network layers perform cross-attention processing based on the same sample data, determine whether the localization hit information of at least one of the cross-attention network layers is a hit; if so, determine that the hit information of the sample data is a hit. If the localization hit information of multiple cross-attention network layers when performing cross-attention processing based on the sample data is all non-hit, then the hit information of the sample data is determined to be non-hit.
[0094] Specifically, for a given sample data, if all the cross-attention network layers show a hit result for that sample data, then the hit information for that sample data pair is considered to be hit.
[0095] If any cross-attention network layer fails to achieve a hit result when performing cross-attention processing on the sample data, it indicates that the hit information corresponding to the sample data is a miss.
[0096] For example, when determining whether the location hit information of multiple cross-attention network layers performing cross-attention processing based on the same sample data is all hit, the location hit information corresponding to the multiple cross-attention network layers can be merged, and the merged value can be compared with a preset sum threshold. If the merged value is greater than or equal to the sum threshold, it indicates that the hit information of the corresponding sample data is hit. If the merged value is less than the sum threshold, it indicates that the hit information of the corresponding sample data is not hit.
[0097] Here, the threshold value is related to the value taken when the hit information of each cross-attention network layer is hit. For example, the value is 1 when the hit information of each cross-attention network layer is hit, and 0 when it is not hit. Since the hit information corresponding to any cross-attention network layer is hit, that is, the hit information of the corresponding sample data is hit, the threshold can be set to 1.
[0098] The merged value is obtained by merging the localization hit information corresponding to multiple cross-attention network layers. For example, it can be represented as: in. Indicates the number of cross-attention network layers; This indicates a merge operation. Indicates the first Location hit information of each cross-attention network layer.
[0099] Then, the number of samples that hit the information is counted, and the ratio of the counted number to the total number of samples is determined as the hit rate.
[0100] A higher hit rate indicates a better localization ability of the visual language model; conversely, a lower hit rate indicates a worse localization ability of the visual language model.
[0101] Alternatively, other methods of determining the hit rate can be used, such as determining the hit rate for each cross-attention network layer. The specific implementation of this disclosure is not limited to these methods.
[0102] Corresponding to the aforementioned embodiments of the visual language model localization capability evaluation method, this application also provides embodiments of a visual language model localization capability evaluation device.
[0103] The embodiments of the visual language model localization capability evaluation device of this application can be applied to computer devices. The device embodiments can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of the computer device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 2 The diagram shown is a hardware structure diagram of a computer device housing the evaluation apparatus for the localization capability of the visual language model of this application. (Except for...) Figure 2 In addition to the processor, memory, network interface, and non-volatile memory shown, the computer device in which the device is located in the embodiment may also include other hardware depending on the actual function of the evaluation device for the localization capability of the visual language model, which will not be described in detail here.
[0104] Please refer to Figure 3 The present disclosure provides an evaluation apparatus for the localization capability of a visual language model, the apparatus comprising: Processing module 31 is used to input sample data into a visual language model to obtain a cross-modal attention tensor output by the cross-modal cross-attention network layer in the visual language model; the sample data includes: images of a graphical user interface and corresponding natural language instructions; the cross-modal attention tensor is used to characterize the distribution of attention to text units corresponding to the natural language instructions and image units corresponding to the images of the graphical user interface. The generation module 32 is used to generate a saliency heatmap of the cross-modal attention network layer based on the cross-modal attention tensor; the saliency heatmap is used to describe the overall visual attention distribution of the image of the graphical user interface; Evaluation module 33 is used to perform location hit determination processing based on the saliency heatmap, and obtain the hit rate of the cross-attention network layer in locating target elements in the graphical user interface during the cross-modal cross-attention processing based on the sample data; the hit rate is used to describe the localization capability of the visual language model.
[0105] Optionally, there may be multiple cross-attention network layers; The generation module 32, when generating the saliency heatmap of the cross-attention network layer based on the cross-modal attention tensor, is used to: The cross-modal attention tensor is split to obtain attention weight matrices corresponding to the multiple cross-attention network layers; Attention weight matrices corresponding to each of the cross-attention network layers are aggregated to obtain aggregated attention weight matrices; and saliency heatmaps corresponding to each of the cross-attention network layers are generated based on the aggregated attention weight matrices.
[0106] Optionally, the attention weight matrix corresponding to each of the cross-attention network layers includes: the attention weight matrix from the text unit to all image units in each of the plurality of attention heads; The generation module 32, when performing attention aggregation processing on the attention weight matrix corresponding to each of the cross-attention network layers to obtain the aggregated attention weight matrix, is used for: The attention weight matrices from text units to all image units corresponding to multiple attention heads are weighted to obtain the aggregated attention weight matrix.
[0107] Optionally, the generation module 32, when generating a saliency heatmap corresponding to each of the cross-attention network layers based on the aggregated attention weight matrix, is used to: The aggregated attention weight matrix is restructured to obtain a two-dimensional spatial structure attention weight matrix; Based on the size of the image corresponding to the graphical user interface, the attention weight matrix of the two-dimensional spatial structure is interpolated to obtain the saliency heatmap corresponding to each of the cross-attention network layers.
[0108] Optionally, the evaluation module 33, when performing location hit determination based on the saliency heatmap and obtaining the hit rate of the cross-attention network layer in locating target elements in the graphical user interface during cross-modal cross-attention processing based on the sample data, is used for: Based on the saliency heatmaps corresponding to each cross-attention network layer and the actual location information of the target element corresponding to the natural language command in the image of the graphical user interface, the localization hit information of each cross-attention network layer when performing cross-attention processing based on the sample data is determined; the localization hit information includes: hit or miss. Based on the localization hit information corresponding to the multiple cross-attention network layers, the hit information of each sample data is determined, and based on the hit information corresponding to the multiple sample data, the hit rate is determined.
[0109] Optionally, the evaluation module 33, when determining the localization hit information of each cross-attention network layer when performing cross-attention processing based on the sample data, according to the saliency heatmaps corresponding to each cross-attention network layer and the real location information of the target element corresponding to the natural language instruction in the image of the graphical user interface, is used to: Extract the coordinates of the point with the maximum response from the saliency heatmap corresponding to each of the cross-attention network layers; Based on the consistency between the position information of the target element in the image of the graphical user interface and the coordinate value of the extracted maximum response point, the hit information of each cross-attention network layer when performing cross-attention processing based on the sample data is determined.
[0110] Optionally, the location information of the target element in the image of the graphical user interface includes: a binary mask image; and the pixel values of each pixel in the binary mask image, used to characterize whether the pixel belongs to the target element. The evaluation module 33, when determining the hit information corresponding to each of the cross-attention network layers based on the consistency between the position information of the target element in the image of the graphical user interface and the coordinate value of the extracted maximum response point, is used to: Based on the extracted coordinates of the maximum response point, the pixel values of the pixels corresponding to the coordinates are read from the binary mask image. Based on the coordinate values of the pixels corresponding to the read coordinate values, the hit information corresponding to each of the cross-attention network layers is determined.
[0111] Optionally, the evaluation module 33, when determining the hit information of each sample data based on the localization hit information corresponding to the multiple cross-attention network layers, is used to: When multiple cross-attention network layers perform cross-attention processing based on the same sample data, determine whether the localization hit information of at least one of the cross-attention network layers is a hit; if so, determine that the hit information of the sample data is a hit. If the localization hit information when performing cross-attention processing based on the sample data in multiple cross-attention network layers is all a miss, then the hit information of the sample data is determined to be a miss.
[0112] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.
[0113] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the visual language model localization capability evaluation method described in the above-described method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.
[0114] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the visual language model localization capability evaluation method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.
[0115] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.
[0116] The computer program or instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions may be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; or an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.
[0117] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0118] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.
Claims
1. A method for evaluating the localization capability of a visual language model, characterized in that, The method includes: The sample data is input into the visual language model to obtain the cross-modal attention tensor output by the cross-modal cross-attention network layer in the visual language model, which corresponds to the sample data. The sample data includes: images of the graphical user interface and corresponding natural language instructions. The cross-modal attention tensor is used to characterize the distribution of attention to text units corresponding to the natural language instructions and image units corresponding to the images of the graphical user interface. Based on the cross-modal attention tensor, a saliency heatmap of the cross-attention network layer is generated; the saliency heatmap is used to describe the overall visual attention distribution to the image of the graphical user interface. Based on the saliency heatmap, a location hit determination process is performed to obtain the hit rate of the cross-attention network layer in locating target elements in the graphical user interface during the cross-modal cross-attention processing based on the sample data; the hit rate is used to describe the localization capability of the visual language model.
2. The method according to claim 1, characterized in that, The cross-attention network layer has multiple layers; The step of generating a saliency heatmap of the cross-modal attention network layer based on the cross-modal attention tensor includes: The cross-modal attention tensor is split to obtain attention weight matrices corresponding to the multiple cross-attention network layers; Attention weight matrices corresponding to each of the cross-attention network layers are aggregated to obtain aggregated attention weight matrices; and saliency heatmaps corresponding to each of the cross-attention network layers are generated based on the aggregated attention weight matrices.
3. The method according to claim 2, characterized in that, The attention weight matrix corresponding to each of the cross-attention network layers includes: the attention weight matrix from the text unit to all image units in each of the multiple attention heads; The step of performing attention aggregation processing on the attention weight matrix corresponding to each of the cross-attention network layers to obtain the aggregated attention weight matrix includes: The attention weight matrices from text units to all image units corresponding to multiple attention heads are weighted to obtain the aggregated attention weight matrix.
4. The method according to claim 2 or 3, characterized in that, The step of generating a saliency heatmap for each of the cross-attention network layers based on the aggregated attention weight matrix includes: The aggregated attention weight matrix is restructured to obtain a two-dimensional spatial structure attention weight matrix; Based on the size of the image corresponding to the graphical user interface, the attention weight matrix of the two-dimensional spatial structure is interpolated to obtain the saliency heatmap corresponding to each of the cross-attention network layers.
5. The method according to claim 1, characterized in that, The location hit determination process based on the saliency heatmap yields the hit rate of the cross-attention network layer in locating target elements in the graphical user interface during cross-modal cross-attention processing based on the sample data, including: Based on the saliency heatmaps corresponding to each cross-attention network layer and the actual location information of the target element corresponding to the natural language command in the image of the graphical user interface, the localization hit information of each cross-attention network layer when performing cross-attention processing based on the sample data is determined; the localization hit information includes: hit or miss. Based on the localization hit information corresponding to the multiple cross-attention network layers, the hit information of each sample data is determined, and based on the hit information corresponding to the multiple sample data, the hit rate is determined.
6. The method according to claim 5, characterized in that, The step of determining the localization hit information of each cross-attention network layer when performing cross-attention processing based on the sample data, according to the saliency heatmaps corresponding to each cross-attention network layer and the real position information of the target element corresponding to the natural language command in the image of the graphical user interface, includes: Extract the coordinates of the point with the maximum response from the saliency heatmap corresponding to each of the cross-attention network layers; Based on the consistency between the position information of the target element in the image of the graphical user interface and the coordinate value of the extracted maximum response point, the hit information of each cross-attention network layer when performing cross-attention processing based on the sample data is determined.
7. The method according to claim 6, characterized in that, The location information of the target element in the image of the graphical user interface includes: a binary mask image; and the pixel value of each pixel in the binary mask image, used to characterize whether the pixel belongs to the target element. The step of determining the hit information corresponding to each of the cross-attention network layers based on the consistency between the position information of the target element in the image of the graphical user interface and the coordinate value of the extracted maximum response point includes: Based on the extracted coordinates of the maximum response point, the pixel values of the pixels corresponding to the coordinates are read from the binary mask image. Based on the coordinate values of the pixels corresponding to the read coordinate values, the hit information corresponding to each of the cross-attention network layers is determined.
8. The method according to claim 6 or 7, characterized in that, The step of determining the hit information of each sample data based on the localization hit information corresponding to the multiple cross-attention network layers includes: When multiple cross-attention network layers perform cross-attention processing based on the same sample data, determine whether the localization hit information of at least one of the cross-attention network layers is a hit; if so, determine that the hit information of the sample data is a hit. If the localization hit information of multiple cross-attention network layers when performing cross-attention processing based on the sample data is all non-hit, then the hit information of the sample data is determined to be non-hit.
9. An evaluation device for the localization capability of a visual language model, characterized in that, The device includes: The processing module is used to input sample data into the visual language model to obtain the cross-modal attention tensor output by the cross-modal cross-attention network layer in the visual language model; the sample data includes: images of the graphical user interface and corresponding natural language instructions; the cross-modal attention tensor is used to characterize the distribution of attention to text units corresponding to the natural language instructions and image units corresponding to the images of the graphical user interface. A generation module is used to generate a saliency heatmap of the cross-modal attention network layer based on the cross-modal attention tensor; the saliency heatmap is used to describe the overall visual attention distribution of the image of the graphical user interface; An evaluation module is used to determine the location hit rate based on the saliency heatmap, and to obtain the hit rate of the cross-attention network layer in locating target elements in the graphical user interface during the cross-modal cross-attention processing based on the sample data; the hit rate is used to describe the localization capability of the visual language model.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1-8.
11. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1-8.