Gui visual positioning attack method and system based on soft coordinate decoding

Abstract

This invention relates to the field of artificial intelligence security and robustness assessment of visual language models. It provides a method and system for adversarial attacks on GUI visual positioning based on soft coordinate decoding. By converting the discrete coordinate digital tokens output by the visual language model into continuously differentiable expected coordinates through soft decoding, and combining decimal position weighting, a loss function is constructed in the coordinate numerical space. This makes the gradient of the adversarial perturbation directly point to the direction that maximizes the coordinate offset, thus realizing efficient adversarial attacks on coordinate numerical space differentiable targets for GUI visual positioning tasks. It can be effectively used for security assessment and robustness testing of GUI intelligent agent systems.

Description

Technical Field

[0001] This invention belongs to the technical field of artificial intelligence security and robustness evaluation of visual language models, and relates to a method and system for GUI visual positioning adversarial attacks based on soft coordinate decoding. Background Technology

[0002] With the development of large-scale visual language models, GUI agents based on VLM models are being widely deployed in scenarios such as browser automation, mobile operation assistance, and enterprise process automation. GUI visual localization is one of its core capabilities: given a GUI screenshot and a natural language instruction (such as "click the confirmation button"), the model outputs the pixel coordinates of the target UI element on the screen.

[0003] Current mainstream GUI localization models (such as UGround and OS-Atlas) adopt a "coordinates as text" output paradigm: screen coordinates are encoded as a sequence of numeric tokens, generated token by token using an autoregressive approach. For example, coordinates (537, 412) are generated as the token sequence "(","5","3","7",",","4","1","2",")". Existing adversarial attack methods targeting GUI visual localization models mainly attack the visual representation space by maximizing the offset of the visual encoder's output features. These attacks are task-independent, do not target the coordinate output structure, and require significant perturbations to the visual representation to indirectly affect the coordinate output, resulting in low efficiency. Since coordinates are output as discrete numeric tokens, and argmax decoding is non-differentiable, standard gradient attack methods cannot directly optimize end-to-end using coordinate offset as the optimization objective. Standard cross-entropy loss assigns the same weight to all numeric tokens, failing to utilize the decimal bit structure of the coordinates. Flipping the hundreds digit of a token results in a shift of hundreds of pixels, while flipping the units digit only results in a shift of a few units. However, equal-weighted CE (Charge-Offset) cannot align the gradient signal to the geometrically most effective direction. Therefore, how to achieve efficient task-aware attacks has become one of the technical problems that need to be solved. Summary of the Invention

[0004] To address the problems existing in the above-mentioned traditional methods, this invention proposes a GUI visual positioning adversarial attack method based on soft coordinate decoding and a GUI visual positioning adversarial attack system based on soft coordinate decoding. It can transform discrete coordinate output into a differentiable optimizable target and realize efficient task perception attack by utilizing the coordinate position value structure.

[0005] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions: On the one hand, a method for GUI visual positioning adversarial attacks based on soft coordinate decoding is provided, including the following steps: From the response token sequence of the visual language model to the input image and the command output, the coordinate axis and decimal place value of each coordinate number token are parsed to generate a set of number token metadata. The discrete probability distribution of digital tokens is transformed into differentiable soft coordinate values ​​by calculating continuous expected values, and then the soft coordinates are obtained by weighting and aggregating the decimal position weights according to the coordinate axes. The attack loss is constructed based on the spatial distance between soft coordinates and real coordinates; With the goal of maximizing attack loss, the adversarial perturbation of the input image is optimized by solving the projection gradient ascent in the coordinate numerical space.

[0006] In one embodiment, the process of parsing the coordinate axis and decimal place value of each coordinate number token includes: Decode the response token sequence into a string, and extract the coordinate number substring using regular expression matching; The decimal place value of each number token is determined based on the actual length of each coordinate number substring, and the absolute position index and coordinate axis of each number token in the complete generated sequence are recorded.

[0007] In one embodiment, the process of converting the discrete digital token probability distribution into differentiable soft coordinate values ​​through continuous expectation calculation includes: For each digital token in the digital token metadata set, obtain the vector of model output logits at the corresponding absolute position; After extracting the logits subvector corresponding to the numeric token identifier from the vector at the corresponding absolute position, the probability distribution of the numeric token at the corresponding absolute position is obtained by normalizing with softmax, and the expected value at the corresponding absolute position is calculated.

[0008] In one embodiment, the process of obtaining soft coordinates by performing decimal positional weight aggregation along the coordinate axes includes: The soft coordinates of the coordinate axes are obtained by summing the expected values ​​of each digit belonging to the same coordinate axis according to their decimal place values.

[0009] In one embodiment, the attack loss is the squared Euclidean distance between the soft coordinates and the true coordinates:

[0010] in, These are the soft coordinate values ​​for the x-axis. These are the soft coordinate values ​​for the y-axis. The true coordinates of the x-axis. These are the actual coordinates of the y-axis.

[0011] In one embodiment, during the process of optimizing the adversarial perturbation of the input image in the coordinate numerical space by projective gradient ascent, the adversarial perturbation is updated in the t-th iteration as follows:

[0012] in, Step size, for Disturb the upper bound of the budget. This indicates that the disturbance will be projected back. Norm constraint domain For the adversarial perturbation in the t-th iteration, The gradient of the attack loss with respect to the perturbation.

[0013] In one embodiment, the step of constructing the attack loss based on the spatial distance between soft coordinates and real coordinates further includes: The attack loss is replaced by the squared Euclidean distance between the soft coordinates and the attacker-specified target coordinates to construct a targeted attack loss; where the targeted attack loss is minimized by gradient descent, the targeted attack loss is:

[0014] in, The x-axis coordinates of the target pixel specified by the attacker. The y-axis coordinate of the target pixel specified by the attacker.

[0015] In one embodiment, the targeted attack optimization process includes: In the language model decoding stage of the visual language model, the tokens at the corresponding positions in the decoded input sequence are forcibly set to a sequence of digital tokens corresponding to the real coordinates, thus constructing a complete differentiable computation graph from the input image to the coordinate token positions logits.

[0016] On the other hand, a GUI visual positioning adversarial attack system based on soft coordinate decoding is also provided, including: The structure parsing module is used to parse the coordinate axis and decimal place value of each coordinate number token from the response token sequence of the visual language model to the input image and the instruction, and generate a set of number token metadata. The soft decoding module is used to convert the discrete probability distribution of digital tokens into differentiable soft coordinate values ​​through continuous expectation value calculation, and then obtain the soft coordinates by performing decimal position weight aggregation according to the coordinate axis. The loss function module is used to construct the attack loss based on the spatial distance between soft coordinates and real coordinates; The gradient solving module is used to optimize the adversarial perturbation of the input image by projecting gradient ascent in the coordinate numerical space with the goal of maximizing the attack loss.

[0017] One of the above technical solutions has the following advantages and beneficial effects: The aforementioned GUI visual positioning adversarial attack method and system based on soft coordinate decoding converts the discrete coordinate digital tokens output by the visual language model into continuously differentiable expected coordinates through soft decoding. Combined with decimal position weighting, a loss function is constructed in the coordinate numerical space, so that the gradient of the adversarial perturbation directly points to the direction of maximizing the coordinate offset. This achieves efficient adversarial attacks on GUI visual positioning tasks with differentiable coordinate numerical space, and can be effectively used for security assessment and robustness testing of GUI intelligent agent systems. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of a GUI visual positioning adversarial attack method based on soft coordinate decoding in one embodiment; Figure 2 This is a diagram illustrating the overall attack method framework of a GUI visual positioning adversarial attack method based on soft coordinate decoding in one embodiment. Figure 3 This is a schematic diagram of the soft coordinate decoding process in one embodiment; Figure 4 This is an attack convergence curve in one embodiment; Figure 5 This is a module architecture diagram of a GUI visual positioning adversarial attack system based on soft coordinate decoding in one embodiment. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention.

[0021] It should be noted that, in this document, the reference to "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The presentation of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will understand that the embodiments described herein can be combined with other embodiments. The term "and / or" as used herein refers to any combination of one or more of the associated listed items, and all possible combinations, including such combinations.

[0022] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0023] The main technical problem addressed in this paper is: in the adversarial robustness evaluation of GUI visual positioning models, how to transform the discrete coordinate digital tokens output by the model into continuously differentiable coordinate representations, and how to construct an attack loss function aligned with the geometric semantics of the coordinates using a decimal positional structure, so that adversarial attacks can directly perform gradient optimization in the coordinate numerical space, achieving a greater reduction in positioning accuracy with fewer iterations under the same perturbation budget.

[0024] The notation conventions are as follows: GUI screenshot: ; Natural language location instructions: ; Actual coordinates: ; The metadata set of coordinate number tokens in the model-generated answer: ,in This represents the absolute position of the token in the sequence. For the corresponding coordinate axis, The decimal place value of the number; Digital token ID (identifier) ​​set: There are 10 in total; Countering disturbances: ,constraint ; Adversarial examples: .

[0025] In one embodiment, such as Figure 1 As shown, a GUI visual positioning adversarial attack method based on soft coordinate decoding is provided, which may include the following processing steps S10 to S16: S10: From the response token sequence of the visual language model to the input image and the instruction output, parse the coordinate axis and decimal place value of each coordinate digital token to generate a set of digital token metadata. S12 transforms the discrete probability distribution of digital tokens into differentiable soft coordinate values ​​through continuous expectation calculation, and then obtains the soft coordinates by performing decimal position weight aggregation according to the coordinate axes; S14, constructing attack loss based on the spatial distance between soft coordinates and real coordinates; S16 aims to maximize the attack loss by optimizing the adversarial perturbation of the input image in the coordinate numerical space through projection gradient ascent.

[0026] It is understood that this embodiment converts the discrete coordinate digital tokens output by the GUI visual positioning model into continuously differentiable expected coordinates through software decoding, and constructs a loss function in the coordinate numerical space using decimal position weighting, so that the gradient against perturbation directly points to the direction that maximizes the coordinate offset. Traditional representation space attacks indirectly affect the coordinate output by maximizing the feature offset in the visual feature space. This is a task-independent attack method that requires significant perturbation of the visual representation to change the coordinate prediction, resulting in low attack efficiency. In contrast, this embodiment directly defines the optimization target in the coordinate numerical space, and the attack target is completely aligned with the positioning task target, achieving a greater reduction in accurate positioning rate under the same perturbation budget.

[0027] Specifically, the GUI visual positioning model encodes screen coordinates into a sequence of numeric tokens, generating them token by token using an autoregressive approach. For example, coordinates (537, 412) are generated as the token sequence "(","5","3","7",",","4","1","2",")". Since the coordinates are output as discrete tokens, the argmax decoding operation is non-differentiable, and standard gradient attack methods cannot directly optimize end-to-end using coordinate offset as the optimization objective. This embodiment uses soft coordinate decoding, instead of directly selecting the number with the highest probability for each numeric token position, it calculates the continuous expected value of that position based on the complete probability distribution output by the model, thereby relaxing the discrete logic space into a continuous coordinate space, enabling gradients to be propagated end-to-end.

[0028] The aforementioned GUI visual positioning adversarial attack method based on soft coordinate decoding converts the discrete coordinate digital tokens output by the visual language model into continuously differentiable expected coordinates through soft decoding. Combined with decimal position weighting, a loss function is constructed in the coordinate numerical space, so that the gradient of the adversarial perturbation directly points to the direction of maximizing the coordinate offset. This achieves efficient adversarial attacks in the coordinate numerical space differentiable for GUI visual positioning tasks and can be effectively used for security assessment and robustness testing of GUI intelligent agent systems.

[0029] In one embodiment, such as Figure 2As shown, regarding step S10 above, the process of parsing the coordinate axis and decimal place value of each coordinate number token may specifically include the following processing: Decode the response token sequence into a string, and extract the coordinate number substring using regular expression matching; The decimal place value of each number token is determined based on the actual length of each coordinate number substring, and the absolute position index and coordinate axis of each number token in the complete generated sequence are recorded.

[0030] It is understandable that, given a model for an input image and instructions The forward inference result is generated by using a known sequence of correct answer tokens as input for each step of the decoding process. The structural information of the coordinate numbers is parsed from the answer token sequence. Specifically, the answer token sequence is decoded into a string, and coordinate number substrings are extracted using regular expression matching based on the coordinate output format of the target model. In this embodiment, the first two consecutive number substrings are extracted as the x-coordinate string and y-coordinate string, respectively. Subsequently, the answer sequence is traversed token by token. For each number token with a value between "0" and "9", its corresponding coordinate axis is determined based on its character offset within the coordinate string. and decimal place value .

[0031] For example, in the three-coordinate string "948", the decimal place value of the first digit "9" is place=100, the decimal place value of the digit "4" is place=10, and the decimal place value of "8" is place=1; for the two-coordinate string "48", the decimal place value of "4" is place=10, and the decimal place value of "8" is place=1. This step outputs a set of metadata for the digit tokens. Each element contains the absolute position index of the token in the complete generated sequence. , corresponding coordinate axis and decimal place value For other coordinate output formats (such as four-axis bounding boxes, y-first then x-first, etc.), the parsing rules can be adjusted accordingly. This embodiment provides the necessary structural information for the subsequent weighted aggregation of coordinate digital tokens by accurately parsing their bit value structure, enabling gradients to be differentiated according to the decimal bit value structure.

[0032] In one embodiment, regarding step S12 above, the process of converting the discrete digital token probability distribution into differentiable soft coordinate values ​​through continuous expectation calculation may specifically include the following processing: For each digital token in the digital token metadata set, obtain the vector of model output logits at the corresponding absolute position; After extracting the logits subvector corresponding to the numeric token identifier from the vector at the corresponding absolute position, the probability distribution of the numeric token at the corresponding absolute position is obtained by normalizing with softmax, and the expected value at the corresponding absolute position is calculated.

[0033] It is understandable that for soft coordinate decoding, such as Figure 3 As shown, the core of this step is to define a differentiable mapping from discrete token probability distributions to continuous coordinate values, enabling the gradient of the subsequent coordinate-level loss function to propagate back to the input image space, fundamentally solving the bottleneck problem of non-differentiability in discrete argmax decoding. For any position on the coordinate axis, instead of directly selecting the digital token with the highest probability (i.e., hard decoding), it is based on the complete probability distribution output by the model. Calculate the continuous expected value of this bit. This allows for differentiable relaxation from discrete logic space to continuous coordinate space.

[0034] Specifically, for the set of metadata for digital tokens Each element in Perform the following calculations: Take the model output logits at the corresponding absolute position The vector at each position is used to extract the logits subvector corresponding to the numeric token identifier (e.g., 10 numeric token IDs). The probability distribution of the numeric value at that position is obtained through softmax normalization, and the expected value at that position is calculated. Each numeric token corresponds to 10 candidate numeric values ​​(0 to 9). The model's output logits at that position is normalized using softmax to obtain a 10-dimensional probability vector. The expected value is equal to the sum of the products of each candidate numeric value and its corresponding probability.

[0035] Among them, the probability distribution of numbers is calculated: ; Calculate the expected value at this location: .

[0036] in, These are candidate numeric values, ranging from 0 to 9. p i ( d ) for this position i Take numbers d The probability of.

[0037] In one embodiment, regarding step S12 above, the process of obtaining soft coordinates by performing decimal positional weight aggregation according to the coordinate axes includes: The soft coordinates of the coordinate axis are obtained by summing the expected values ​​of each digit belonging to the same coordinate axis according to their decimal place values. Taking the x-axis as an example:

[0038] Right now (Taking a three-digit coordinate system as an example). Soft coordinates of the y-axis. Calculate in the same way.

[0039] In this embodiment, the positional weighting mechanism aggregates the expected values ​​of different digits according to their actual pixel contribution: the expected value of the hundreds digit is multiplied by 100, the tens digit by 10, and the units digit by 1. The aggregated result directly corresponds to continuous coordinate values ​​in the pixel space. This ensures that the gradient of the subsequent loss function can be automatically allocated according to the actual contribution ratio of each digit to the coordinate offset. The hundreds digit token receives a gradient weight that is much greater than that of the units digit token, so that the optimization process automatically focuses on the higher coordinates that are most critical to the attack effect.

[0040] In one embodiment, the attack loss is the squared Euclidean distance between the soft coordinates and the true coordinates:

[0041] in, These are the soft coordinate values ​​for the x-axis. These are the soft coordinate values ​​for the y-axis. The true coordinates of the x-axis. These are the actual coordinates of the y-axis.

[0042] Specifically, the attack loss is defined as the squared Euclidean distance between the soft coordinates and the true coordinates, and the attack target is to maximize... That is, to find adversarial perturbations that make the model's predicted coordinates as far away from the true coordinates as possible.

[0043] The advantage of this loss function compared to the standard cross-entropy loss is that the CE loss assigns the same gradient weight to both "hundreds digit changing from 5 to 6" and "units digit changing from 7 to 8," but the former results in a 100-pixel shift while the latter only results in a 1-pixel shift. (Coordinates in this embodiment) The loss function automatically weights the gradient assignments proportionally to the pixel offset, ensuring the optimization process naturally focuses on the high-order digit token that contributes the most to the coordinate offset. From an optimization perspective, this loss function directly defines the attack target in the coordinate numerical space, perfectly aligning with the evaluation metrics for GUI localization tasks and avoiding the task misalignment problem in cross-entropy loss.

[0044] In one embodiment, regarding step S16 above, during the process of optimizing the adversarial perturbation of the input image through projection gradient ascent in the coordinate numerical space, in the t-th iteration, the adversarial perturbation is updated according to the following formula:

[0045] in, Step size, for Disturb the upper bound of the budget. This indicates that the disturbance will be projected back. Norm constraint domain For the adversarial perturbation in the t-th iteration, The gradient of the attack loss with respect to the perturbation.

[0046] Specifically, for the ascent solution of the projected gradient, initialization is performed against perturbations. It is random noise. Iterative execution: (1) Constructing adversarial examples:

[0047] (2) To maintain normal operation of the visual encoder, in the language model decoding stage, in order to construct a complete differentiable computation graph from the image to the coordinate token position logits, the historical tokens at the corresponding positions in the decoded input sequence are forcibly set to the same as the real coordinates. The corresponding sequence of digital tokens (i.e., teacher forcing). This operation ensures coordinate space loss. The gradient can be stably backpropagated to the image space.

[0048] (3) Calculate the soft coordinates according to the parsed digital token structure above. .

[0049] (4) Calculate the loss The gradient is obtained through backpropagation. .

[0050] (5) Update the perturbation and project it back to the constraint domain. The total number of iterations is T.

[0051] This iterative optimization strategy of successive projections ensures that the magnitude of the perturbation never exceeds the preset budget, while each step updates along the direction of the fastest increase in loss, ensuring attack efficiency.

[0052] In one embodiment, regarding step S14 above, the step of constructing the attack loss based on the spatial distance between the soft coordinates and the true coordinates may further include the following processing: The attack loss is replaced by the squared Euclidean distance between the soft coordinates and the attacker-specified target coordinates to construct a targeted attack loss; where the targeted attack loss is minimized by gradient descent, the targeted attack loss is:

[0053] in, The x-axis coordinates of the target pixel specified by the attacker. The y-axis coordinate of the target pixel specified by the attacker.

[0054] It is understandable that for targeted attacks, the loss function can be replaced with a targeted version based on the non-targeted attack, and gradient descent can be used to minimize the soft coordinates to the attacker's specified target location. The distance. For example, the target can be set as the coordinates of a dangerous button on the screen, allowing the model's click to be hijacked from a safe location to the attacker's desired location.

[0055] This targeted attack capability is a natural extension of the coordinate space loss function: due to soft coordinates Directly corresponding to two-dimensional continuous values ​​in pixel space, targeted attacks can construct corresponding directional losses by specifying arbitrary target coordinates without requiring additional method design. Compared to non-directional attacks that only seek to deviate from real coordinates, targeted attacks can be used to evaluate hijacking attack scenarios that pose a more realistic threat, thus having higher security assessment value.

[0056] In one embodiment, the targeted attack optimization process may further include the following steps: In the language model decoding stage of the visual language model, the tokens at corresponding positions in the decoded input sequence are forcibly set to a sequence of numeric tokens corresponding to the real coordinates (i.e., teacher forcing). This constructs a complete differentiable computational graph from the input image to the logits at the coordinate token positions, ensuring that the gradient of the coordinate space loss is stably backpropagated to the input image space. This teacher forcing mechanism addresses the incomplete gradient backpropagation path during the autoregressive decoding process of the visual language model: in normal autoregressive generation, the input token of the current step depends on the sampling result of the previous step, and the sampling operation is non-differentiable, severing the gradient link from the output logits to the input image. By forcibly replacing the tokens at the coordinate numeric token positions in the decoded input sequence with the numeric tokens corresponding to the real coordinates, a complete differentiable computational graph from image encoding and language model decoding to the logits at the coordinate token positions is constructed while maintaining normal visual encoder operation, ensuring that the gradient of the coordinate space loss can be stably backpropagated to the input image. It is worth noting that this mechanism is only used during adversarial perturbation optimization and does not change the model's inference behavior itself.

[0057] In some implementations, experimental examples are also provided: The aforementioned GUI visual localization adversarial attack method based on soft coordinate decoding was employed to verify its effectiveness against two target models with different architectures in various GUI environments. The specific experimental configuration is as follows: Dataset: The existing ScreenSpot-V2 test set was used for evaluation. 50 samples were randomly selected from each of the three types of GUI environments: mobile, desktop, and web, for a total of 150 test samples, covering two types of target UI elements: text and icon.

[0058] Target Models: Two representative GUI visual positioning models were selected for verification: (1) OS-Atlas-Base-7B: based on the Qwen2-VL architecture, using a coordinate text token autoregressive output paradigm; (2) UGround-V1-7B: based on the LLaVA architecture, also using a coordinate text token autoregressive output paradigm, but the underlying architecture and word segmenter are different from OS-Atlas. The selection of two models with different architectures aims to verify the architectural universality of the method of this invention.

[0059] Attack parameters: Disruption Budget Step length Number of iterations The above parameter settings are consistent with the baseline papers to ensure a fair comparison.

[0060] Compared to baseline methods: (1) Representation Attack: a non-directional attack method that maximizes the difference between the output features of the visual encoder and the original features. Distance-based adversarial examples are constructed. The optimization objective is... ,constraint This method only requires access to the visual encoder, making it a task-independent attack method.

[0061] (2) Standard Token-level CE attack: PGD attack that uses the negative log-likelihood (Cross-Entropy) of the real coordinate numeric token as the loss function. This method assigns the same gradient weight to all numeric token positions and does not utilize the decimal bit value structure of the coordinates.

[0062] The evaluation metric is the post-attack localization accuracy (ACC, sorted in descending order): the model's success rate in localizing adversarial examples; the lower the ACC, the stronger the attack effect. The criteria for determining successful localization are consistent with the original ScreenSpot-V2 evaluation protocol, i.e., the predicted coordinates falling within the bounding box of the target UI element are considered successful.

[0063] Table 1 shows a comparison of the effectiveness of different attack methods on OS-Atlas-Base-7B and UGround-V1-7B: Table 1

[0064] The following results can be observed from Table 1: (1) The above-mentioned GUI visual positioning adversarial attack method based on soft coordinate decoding (hereinafter referred to as the method of this invention) achieved significantly better attack performance than the baseline method on target models with two different architectures. On OS-Atlas-Base-7B, the method of this invention reduced the average positioning accuracy from 0.81 in the no-attack state to 0.18, while the representation space attack only reduced it to 0.37, and the Token CE attack only reduced it to 0.45. On UGround-V1-7B, the method of this invention reduced the average accuracy from 0.90 to 0.20, while the representation space attack only reduced it to 0.53, and the Token CE attack only reduced it to 0.71.

[0065] (2) The effectiveness ranking of the three attack methods is completely consistent across the two models: the method of this invention is superior to the representation space attack, and the representation space attack is superior to the Token CE attack. This consistency indicates that the advantage of the method of this invention does not depend on a specific model architecture and has good versatility.

[0066] (3) Token CE attack performed the worst, especially on UGround-V1-7B, where it only reduced the accuracy from 0.90 to 0.71. This verifies the problem described in the background: standard CE loss assigns the same weight to all digital token positions. Flipping the hundreds digit causes a 100-pixel shift, while flipping the units digit only causes a 1-pixel shift. However, equal-weighted CE cannot distinguish between the two, resulting in a dispersed gradient signal and low attack efficiency.

[0067] (4) While representation space attacks are superior to CE attacks, they are still significantly weaker than the method of this invention. Representation space attacks indirectly affect coordinate output by maximizing visual feature offsets. This is a task-independent attack method that requires a significant perturbation to the visual representation to change the coordinate prediction. In contrast, the method of this invention directly constructs a loss function in the coordinate numerical space. The attack target is perfectly aligned with the localization task target, thus achieving a greater decrease in accuracy under the same perturbation budget.

[0068] (5) The method of the present invention is stable in different GUI environments. In the OS-Atlas-Base-7B Desktop and Web environments, the accuracy decreased from 0.86 and 0.78 to 0.12, respectively, with an absolute decrease of 0.74 and 0.66. In the UGround-V1-7B Web environment, the accuracy decreased from 0.90 to 0.14, with an absolute decrease of 0.76.

[0069] The necessity of decimal positional weighting was also verified through ablation experiments. As shown in Table 2, the "equal-weight aggregation" variant in the ablation experiments replaced the coordinate space loss function with the sum of positional mean square errors, i.e. ,in Let i be the expected value for the soft decoding of the i-th bit. The value (0 to 9) is the numeric value of the true coordinate at that position. This variant assigns the same gradient weight to each digit, aggregates the coordinate-level loss without decimal position weights, and the rest of the steps are exactly the same as the full method.

[0070] Table 2

[0071] As shown in Table 2, removing the decimal place value weighting significantly reduces the attack effectiveness. On OS-Atlas-Base-7B, the average accuracy of the equal-weighted aggregation variant is 0.48, while the full method is 0.18, a difference of 0.30. On UGround-V1-7B, the equal-weighted aggregation variant achieves 0.43, while the full method achieves 0.20, a difference of 0.23. Both models exhibit a consistent trend, indicating that the place value weighting mechanism is the key source of the attack effectiveness of the method in this invention. Equal-weighted aggregation assigns the same gradient weights to the hundreds, tens, and units digits, failing to focus the optimization signal on the high-order digit token that contributes the most to the coordinate offset. This leads to gradient dispersion and attack efficiency degradation, essentially the same problem faced by Token CE attacks.

[0072] The horizontal axis represents the number of iterations (0 to 100), and the vertical axis represents the average localization accuracy (ACC, sorted in descending order). The average accuracy (ACC, sorted in descending order) is the macro-average of the OS-Atlas-Base-7B and UGround-V1-7B models across three types of GUI environments. The two curves represent the method of this invention and the attack on the characterization space, respectively, averaged across the OS-Atlas-Base-7B and UGround-V1-7B models in three types of GUI environments. The shaded area represents the standard deviation. Figure 4As shown in the attack convergence curve, the method of this invention reaches the accuracy level of characterizing spatial attacks at step 100 (shown by the dashed line) around step 15, with a convergence speed approximately 6 times faster. In the final state, the method of this invention reduces the accuracy to 0.19, and the accuracy for characterizing spatial attacks is 0.45.

[0073] This invention achieves a coordinate numerical space differentiability adversarial attack against GUI visual positioning tasks through soft coordinate decoding and decimal position weighting. Overall, compared to existing technologies: (1) This invention solves the problem of non-differentiability of discrete coordinate digital token output, enabling gradients to be backpropagated from the coordinate numerical space to the input image end-to-end, providing an effective tool for evaluating the adversarial robustness of GUI localization models. On OS-Atlas-Base-7B, the method of this invention reduces the average accuracy from 0.81 to 0.18 (a reduction of 0.63), far exceeding the reductions of representation space attacks (0.44) and Token CE attacks (0.36). On UGround-V1-7B, the method of this invention reduces the average accuracy from 0.90 to 0.20 (a reduction of 0.70), far exceeding the reductions of representation space attacks (0.37) and Token CE attacks (0.19).

[0074] (2) The position weighting mechanism enables the gradient to automatically focus on the high-order digit token that contributes the most to the coordinate offset. The Token CE attack assigns equal weight gradients to all digits, which only reduces the accuracy by 0.19 on UGround-V1-7B, while the method of this invention reduces it by 0.70, with a difference of 3.7 times, which fully verifies the key role of position weighting in gradient semantic alignment.

[0075] (3) The convergence speed of the method of this invention is significantly better than that of characterization space attacks. (From the appendix...) Figure 4 As can be seen, the method of the present invention achieves the accuracy level of characterizing space attacks at step 100 in about step 15, and the convergence speed is about 6 times that of the latter, which can significantly reduce the number of iterations and computation required for the attack.

[0076] (4) This method naturally supports targeted attacks (specifying target coordinates). By simply replacing the loss function with the L2 distance from the soft coordinates to the target coordinates and using gradient descent, the model click can be hijacked from a safe location to the attacker's desired location. This method can be used to evaluate hijacking attack scenarios that pose a more realistic threat.

[0077] (5) This method is applicable to all GUI positioning models that use coordinate text token output paradigms, without depending on a specific model architecture. Experiments on OS-Atlas-Base-7B based on Qwen2-VL architecture and UGround-V1-7B based on LLaVA architecture have verified the effectiveness of the method, and the effect ranking of the three attack methods is completely consistent on the two models, indicating that the advantage of this invention is its architectural universality.

[0078] It should be understood that, although Figure 1 The steps are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order in which these steps are executed; they can be performed in other orders. Figure 1 At least some of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0079] In one embodiment, such as Figure 5 As shown, a GUI visual positioning adversarial attack system based on soft coordinate decoding is also provided, which may include a structure parsing module 11, a soft decoding module 13, a loss function module 15, and a gradient solving module 17. The structure parsing module 11 is used to parse the coordinate axis and decimal place value of each coordinate digital token from the response token sequence of the visual language model to the input image and command output, generating a set of digital token metadata. The soft decoding module 13 is used to convert the discrete digital token probability distribution into differentiable soft coordinate values ​​through continuous expectation calculation, and then perform decimal place value weighted aggregation according to the coordinate axis to obtain the soft coordinates. The loss function module 15 is used to construct the attack loss based on the spatial distance between the soft coordinates and the real coordinates; the gradient solving module 17 is used to optimize the adversarial perturbation of the input image in the coordinate numerical space by projecting gradient ascent with the goal of maximizing the attack loss.

[0080] It is understood that the specific limitations of each feature in the above-mentioned GUI visual positioning adversarial attack system based on soft coordinate decoding can be found in the explanation of the corresponding features in the various embodiments of the GUI visual positioning adversarial attack method based on soft coordinate decoding above, and will not be repeated here.

[0081] The aforementioned GUI visual positioning adversarial attack system based on soft coordinate decoding converts the discrete coordinate digital tokens output by the GUI visual positioning model into continuously differentiable expected coordinates through soft decoding. Combined with decimal position weighting, a loss function is constructed in the coordinate numerical space, so that the gradient of the adversarial perturbation directly points to the direction that maximizes the coordinate offset. This achieves efficient adversarial attacks on GUI visual positioning tasks with differentiable coordinate numerical space, and can be effectively used for security assessment and robustness testing of GUI intelligent agent systems.

[0082] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM). By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus type dynamic random access memory (Rambus DRAM, or RDRAM for short), and direct Rambus DRAM (DRDRAM), etc.

[0083] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0084] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of protection of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and all such modifications and improvements fall within the scope of protection of the present invention.

Claims

1. A GUI visual positioning adversarial attack method based on soft coordinate decoding, characterized in that, Including the following steps: From the response token sequence of the visual language model to the input image and the command output, the coordinate axis and decimal place value of each coordinate number token are parsed to generate a set of number token metadata. The discrete probability distribution of digital tokens is transformed into differentiable soft coordinate values ​​by calculating continuous expected values, and then the soft coordinates are obtained by weighting and aggregating the decimal position weights according to the coordinate axes. The attack loss is constructed based on the spatial distance between the soft coordinates and the real coordinates; With the goal of maximizing the attack loss, the adversarial perturbation of the input image is optimized by solving the projection gradient ascent in the coordinate numerical space.

2. The GUI visual positioning adversarial attack method based on soft coordinate decoding according to claim 1, characterized in that, The process of parsing the coordinate axis and decimal place value of each coordinate number token includes: Decode the response token sequence into a string, and extract the coordinate number substring using regular expression matching; The decimal place value of each number token is determined based on the actual length of each coordinate number substring, and the absolute position index and coordinate axis of each number token in the complete generated sequence are recorded.

3. The GUI visual positioning adversarial attack method based on soft coordinate decoding according to claim 1, characterized in that, The process of transforming the discrete probability distribution of digital tokens into differentiable soft coordinate values ​​through continuous expectation calculation includes: For each digital token in the digital token metadata set, obtain the vector of model output logits at the corresponding absolute position; After extracting the logits subvector corresponding to the numeric token identifier from the vector at the corresponding absolute position, the probability distribution of the numeric token at the corresponding absolute position is obtained by normalizing with softmax, and the expected value at the corresponding absolute position is calculated.

4. The GUI visual positioning adversarial attack method based on soft coordinate decoding according to claim 3, characterized in that, The process of obtaining soft coordinates by performing decimal positional weight aggregation according to the coordinate axes includes: The soft coordinates of the coordinate axes are obtained by summing the expected values ​​of each digit belonging to the same coordinate axis according to their decimal place values.

5. The GUI visual positioning adversarial attack method based on soft coordinate decoding according to claim 4, characterized in that, The attack loss is the squared Euclidean distance between the soft coordinates and the true coordinates: in, These are the soft coordinate values ​​for the x-axis. These are the soft coordinate values ​​for the y-axis. The true coordinates of the x-axis. These are the actual coordinates of the y-axis.

6. The GUI visual positioning adversarial attack method based on soft coordinate decoding according to claim 5, characterized in that, In the process of optimizing the adversarial perturbation of the input image through projection gradient ascent in the coordinate numerical space, the adversarial perturbation is updated in the t-th iteration as follows: in, Step size, for Disturb the upper bound of the budget. This indicates that the disturbance will be projected back. Norm constraint domain For the adversarial perturbation in the t-th iteration, The gradient of the attack loss with respect to the perturbation.

7. The GUI visual positioning adversarial attack method based on soft coordinate decoding according to claim 5, characterized in that, The step of constructing the attack loss based on the spatial distance between the soft coordinates and the real coordinates also includes: The attack loss is replaced by the squared Euclidean distance between the soft coordinates and the attacker-specified target coordinates to construct a targeted attack loss; wherein, the targeted attack loss is minimized by gradient descent, and the targeted attack loss is: in, The x-axis coordinates of the target pixel specified by the attacker. The y-axis coordinate of the target pixel specified by the attacker.

8. The GUI visual positioning adversarial attack method based on soft coordinate decoding according to claim 7, characterized in that, The optimization process for targeted attacks includes: In the language model decoding stage of the visual language model, the tokens at the corresponding positions in the decoded input sequence are forcibly set to a sequence of digital tokens corresponding to the real coordinates, thus constructing a complete differentiable computation graph from the input image to the coordinate token positions logits.

9. A GUI visual positioning adversarial attack system based on soft coordinate decoding, characterized in that, include: The structure parsing module is used to parse the coordinate axis and decimal place value of each coordinate number token from the response token sequence of the visual language model to the input image and the instruction, and generate a set of number token metadata. The soft decoding module is used to convert the discrete probability distribution of digital tokens into differentiable soft coordinate values ​​through continuous expectation value calculation, and then obtain the soft coordinates by performing decimal position weight aggregation according to the coordinate axis. The loss function module is used to construct the attack loss based on the spatial distance between the soft coordinates and the real coordinates; The gradient solving module is used to optimize the adversarial perturbation of the input image by projecting gradient ascent in the coordinate numerical space with the goal of maximizing the attack loss.

10. The GUI visual positioning adversarial attack system based on soft coordinate decoding according to claim 9, characterized in that, The process of parsing the coordinate axis and decimal place value of each coordinate number token includes: Decode the response token sequence into a string, and extract the coordinate number substring using regular expression matching; The decimal place value of each number token is determined based on the actual length of each coordinate number substring, and the absolute position index and coordinate axis of each number token in the complete generated sequence are recorded.

Images