Advertisement interference evaluation benchmark automatic construction method for mobile terminal graphical user interface agent

Abstract

The invention discloses a mobile terminal graphical user interface agent-oriented advertisement interference evaluation benchmark automatic construction method, which comprises the following steps of: carrying out abstract modeling on a mobile terminal interface structure and control semantics, and collecting static group and dynamic group data; automatically injecting multiple types of advertisement elements into the insertable area, and generating matched samples of an advertisement version and a non-advertisement version; an advertisement type, a closing control position and the like are automatically identified through a multi-modal detection and interface alignment algorithm, a reproducible behavior event file is generated by utilizing an action sequence alignment module, and accurate mapping of task semantics and interference behaviors is realized; and generating evaluation index data including indexes such as a task completion rate, a misoperation rate and an advertisement closing accuracy rate, and realizing repeatable evaluation of the robustness of the interface agent or the large language model in the advertisement interference environment. The method has the characteristics of automation, expandability and high universality, and a unified technical reference is provided for robustness evaluation and optimization of the intelligent agent.

Description

Technical Field

[0001] This invention relates to the field of intelligent human-computer interaction and automated testing, and in particular to an automatic method for constructing an evaluation benchmark for advertising interference in mobile graphical user interface intelligent agents. Background Technology

[0002] With the continuous development of mobile internet applications, GUI agents have been widely used in mobile automated testing, intelligent operation and maintenance, accessibility assistance, and human-computer interaction. These agents typically parse interface structures, identify control elements, and combine language models or reinforcement learning algorithms to complete interactive operations, achieving automated task execution in multiple scenarios. In recent years, with the emergence of Multimodal Large Models (MLLM), the understanding and decision-making capabilities of GUI agents have been significantly enhanced; however, their stability and generalization ability in complex real-world environments still face challenges.

[0003] For example, patent document CN119576470A discloses a model construction method for GUI intelligent agents. By automatically annotating the task execution trajectory, it guides a large model to actively annotate and understand the thinking process in human operations, and further plan the next action. Patent document CN120375152A discloses an automatic generation method for GUI view operation state machines, which improves the ability of intelligent agents to handle complex GUI interaction commands.

[0004] However, in the actual operation of mobile applications, various advertising elements (including pop-up ads, banner ads, interstitial ads, and rewarded video ads) frequently interfere with interface interaction, seriously affecting the agent's task judgment and execution path. The randomness, dynamism, and diversity of advertisements make it easy for the agent to misidentify, misclick, and deviate from the path when identifying key controls, locating target areas, and planning operation sequences, leading to task failure or interruption. Existing research mostly focuses on idealized, interference-free environments, and the modeling, evaluation, and optimization of advertising interference factors are still significantly insufficient.

[0005] Currently, there is a lack of standardized datasets and evaluation benchmarks for systematically measuring the impact of ad interference. On the one hand, manually constructing ad scenarios is costly and limited in scale, making it difficult to truly reflect diverse application environments. On the other hand, existing automated data collection methods are mostly aimed at functional testing, failing to create a comparable data structure with and without ads, and also lacking a unified indicator system and automated evaluation platform. Therefore, there is an urgent need for an automated construction method that can automatically generate ad interference samples, build standardized evaluation datasets, and support unified indicator quantification and multi-agent comparison, in order to promote the robustness evaluation and optimization research of GUI agents in real-world complex scenarios. Summary of the Invention

[0006] To address the lack of real-world advertising interference data, a unified evaluation environment, and a systematic indicator system in existing technologies, this invention provides an automatic construction method for advertising interference evaluation benchmarks for mobile graphical user interface (GUI) smart agents. This method enables standardized automatic construction of advertising interference samples and multi-dimensional performance evaluation, providing a unified data foundation and technical support for the robustness analysis and optimization of smart agents in complex mobile environments.

[0007] An automatic construction method for advertising interference evaluation benchmarks for mobile graphical user interface intelligent agents includes: S1, based on the interface hierarchy tree parsing algorithm, abstracts and models the interface structure and control semantics of mobile applications, and collects and forms static group data and dynamic group data. S2 injects multiple types of advertising elements into the interface of static group data through template matching and generative advertising synthesis model; and collects the real advertising display process in dynamic group data through integrated advertising SDK; finally, it obtains static no-ad samples and paired static with-ad samples, dynamic no-ad samples and paired dynamic with-ad samples. S3 employs a multimodal detection and interface alignment algorithm to automatically identify the advertising area, advertising type, close control position, and advertising display duration in static and dynamic advertising samples, and outputs a unified structured annotation file, maintaining a correspondence with the task events of the sample on the timeline; S4, using the action sequence alignment module, inserts advertising processing actions into task events with static and dynamic ad samples, generating corresponding behavior event files; S5. The static group data and dynamic group data processed in steps S2 to S4 are used to construct an ad interference dataset. The task number, ad type and ad location distribution are recorded through a metadata index file to realize the mapping between task semantics and ad interference behavior. S6 generates evaluation metrics based on the advertising interference dataset, including task completion rate, misoperation rate, advertising detection accuracy, path recovery rate, and response latency, to quantify the impact of advertising interference on agent performance. S7 connects the aforementioned ad interference dataset to the testing platform through a unified interface to achieve robustness comparison and automated evaluation of different graphical user interface agents or multimodal language models.

[0008] In step S1, during the process of collecting and forming static group data, only file reading and metadata organization are performed, without modifying the task logic or interface content, to ensure the structural integrity and reproducibility of the data.

[0009] In step S2, the actual ad display process is collected in the dynamic group data by integrating the ad SDK. Specifically, this includes: embedding the ad SDK into the mobile application to generate an application with ads; running the ad-free and ad-ad applications on real or virtual devices respectively; fully capturing the entire process of ad loading, display, and closing; and saving the corresponding interface screenshots, interface structure files, and event logs.

[0010] In step S3, the multimodal detection and interface alignment algorithm adopts a combination of visual detection and XML hierarchical analysis. The Vision-XML Alignment module simultaneously identifies the ad area, ad type, close control position, and ad display duration, and outputs a unified structured annotation file.

[0011] In step S4, ad processing actions are inserted into both static and dynamic task events containing ad samples, specifically as follows: For splash screen ads, generate an ad processing action: detect the ad, wait for it to close, click to close, and then continue the task. For interstitial ads, generate an ad processing action that involves an operation chain of detecting the ad, clicking to close, and continuing the task. For banner ads, generate an ad processing action sequence of events: detect the ad, avoid the ad area, and continue the task. For incentivized video ads, generate an ad processing action that involves detecting the ad, playing the ad, waiting for the ad to end, and then continuing the task. For native ads, generate ad processing actions that identify ad content, adjust interactive focus, and continue the task.

[0012] In step S5, the ad interference dataset is indexed by the task number of the sample. Each task number contains both ad-free and ad-containing samples in its corresponding directory. The two samples are consistent in terms of task objectives, interaction logic, and file structure.

[0013] In step S5, during the construction of the ad interference dataset, samples that do not meet the alignment criteria are automatically filtered out by performing anomaly detection and resampling on the task execution logs and ad interference intensity.

[0014] In step S7, the testing platform includes a task execution interface and an evaluation metric calculation module; The task execution interface supports different graphical user interface agents or multimodal language models to automatically execute tasks in the same dataset environment; the evaluation index calculation module automatically generates task completion rate, misoperation rate, ad detection accuracy, path recovery rate and response latency based on the event logs and annotation files during the execution of sample task events, and outputs evaluation report files in a unified format.

[0015] Compared with the prior art, the present invention has the following beneficial effects: This invention realizes a complete process of automatically constructing benchmark data, generating indicators, and conducting platform-based evaluation for advertising interference assessment. The method, through a dual-structure design of static and dynamic groups, balances data controllability and authenticity. It can automatically generate paired samples with and without ads and complete unified annotation and indicator calculation, thereby significantly reducing manual processing costs, improving data consistency and experimental reproducibility. This provides systematic and scalable technical support for the robustness evaluation and optimization of multimodal large language models and graphical user interface agents. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the construction process of the advertising interference dataset in an embodiment of the present invention.

[0018] Figure 2 This is a schematic diagram illustrating the construction and evaluation analysis of the unified testing platform in this embodiment of the invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0021] An automatic construction method for advertising interference evaluation benchmarks for mobile graphical user interface intelligent agents mainly includes the construction of advertising interference datasets, the construction of a unified testing platform, and evaluation analysis.

[0022] like Figure 1 As shown, the construction of the advertising interference dataset mainly includes three stages: data collection, data processing, and dataset integration, forming a standardized and reproducible evaluation dataset.

[0023] S10. Data Acquisition Phase. This phase is used to establish the basic data resources required for advertising interference evaluation, and is divided into static and dynamic groups. The system adopts automated acquisition scripts and a multi-threaded control mechanism to achieve efficient, stable, and batch sample acquisition.

[0024] S101. Static Group Data Acquisition. Static group data is based on existing publicly available GUI datasets (such as LLaMATouch, AndroidLab, etc.). An automated script batch-reads interface screenshots (.jpg), interface hierarchy structure files (XML dump), task description files (instruction.txt), and operation sequence files (events.txt) from the task directory. The collected samples are all ad-free versions, used for subsequent ad insertion and sample generation with ads. Static groups support three ad types: splash screen ads, interstitial ads, and banner ads. During the acquisition process, the system only reads files and organizes metadata, without modifying task logic or interface content, ensuring the structural integrity and reproducibility of the data. The advantages of static groups are strong controllability, stable sample structure, and ease of conducting large-scale control experiments and algorithm testing.

[0025] S102. Dynamic Group Data Acquisition. Dynamic group data is collected in a real mobile application environment to reflect the true characteristics of ad loading and interaction. The system selects an integrable APP project, embeds an advertising SDK (such as Google AdMob, Pangle, Youlianghui, etc.) into it, and generates an application with ads. Through automated task scripts, the system runs the ad-free and ad-enabled versions of the application on real or virtual devices, executes task instructions, and simultaneously collects frame screenshot sequences, interface structure files, and behavior logs. The ad types included in the dynamic group are not only splash screens, interstitial screens, and banners, but also native ads and rewarded video ads. During the acquisition phase, the system performs multi-source data recording, timestamp marking, and synchronous storage operations to completely capture the entire process of ad loading, display, and closing, and saves corresponding screenshots, structure files, and event logs, providing rich raw input for subsequent data processing stages. The advantage of dynamic groups is that they closely resemble real user interaction scenarios and can reflect the temporal changes and interactive complexity of ads under different device and system environments.

[0026] The combination of static and dynamic data sets forms a "controllable + realistic" dual-modal acquisition system: the static set provides a stable, structured baseline for accurately evaluating the agent's performance under ideal conditions; the dynamic set supplements real-world operational data to characterize the agent's robustness in complex environments. This combination ensures that the dataset of this invention is both reproducible and realistically representative, providing highly reliable data support for subsequent evaluation platforms.

[0027] S20. Data Processing Stage. This stage is the core of the invention, used to transform the collected samples into a standardized, comparable dataset for evaluating advertising interference. The system sequentially executes processes such as data parsing, ad injection, annotation alignment, behavioral event generation, and data expansion, ultimately forming a dataset that can be directly used for performance analysis.

[0028] S201. Data Parsing and Formatting. The system performs unified formatting and path management on the collected static and dynamic group data. For static group samples, the system parses the interface structure file, simplifying the XML file into JSON format, retaining only the core fields related to task semantics, spatial layout, and operational logic. For dynamic group samples, the system parses the collected log files and screenshot sequences, extracting information such as task boundaries, ad appearance times, and interaction trajectories. The system uses a multimodal large language model (MLLM) to jointly analyze the log text and interface screenshots, automatically generating a task description file (instruction.txt), completing missing task semantics, and standardizing field naming conventions to ensure consistent structure and parsability of data from different sources.

[0029] S202. Ad Generation and Injection. The system injects ad elements into static group interface screenshots or structure files based on template matching methods and a real ad creative library. The ad creative library includes various ad creative types such as splash screens, interstitial screens, banners, native ads, and rewarded video ads. Each creative includes information on resolution, aspect ratio, display rules, and closing method, and conforms to the display specifications of mainstream advertising platforms. The system automatically selects the ad type and insertion position based on the interface layout, task semantics, and control distribution, achieving natural visual integration and logical consistency.

[0030] The system employs an intelligent ad insertion strategy, comprehensively considering interface complexity, interaction path density, and user attention prediction results to automatically determine the quantity, location, and format of ads. For example, short tasks automatically insert splash screen ads; medium tasks insert interstitial or banner ads; and complex tasks dynamically insert multiple types of ads at different interaction stages to enhance the interference intensity. The system utilizes a visual saliency analysis algorithm to locate non-task-critical areas, ensuring that ads both interfere and do not disrupt the task structure. After injection, the system generates an ad injection log file, recording the ad type, display location, duration, and closing method, providing basic data for subsequent annotation and behavioral event generation.

[0031] S203. Ad Labeling and Behavioral Event Generation. After ad injection, the system combines visual detection and structural matching algorithms to identify the ad area and the location of interactive controls. At the visual layer, the system uses boundary detection, template matching, and text recognition methods to extract the ad bounding box and the coordinates of the close button; at the structural layer, the system uses node matching methods to map the ad controls to the interface hierarchy. The generated label file ads.json records information such as ad area coordinates, type, close control location, and display duration, and maintains a correspondence with task events on the timeline.

[0032] When generating a task with ads, the system copies and expands the event file of the ad-free task. Combining the original task action sequence, the system automatically inserts ad-related operation instructions into the new event file. For example: For splash screen ads, the system generates an ad processing action of "detect ad - wait to close - click close - continue task"; For interstitial ads, the system generates an ad processing action of "detect ad - click close - continue task"; For banner ads, the system generates an ad processing action of "detect ad - avoid ad area - continue task"; For incentivized video ads, the system generates an ad processing sequence of "detect ad - play ad - wait for ad to end - continue task"; For native ads, the system generates an ad processing action of "identify ad content - adjust interactive focus - continue task".

[0033] The system synchronously updates the control triggering order and node hierarchy in the interface structure file during the instruction generation process to ensure that the process of ad appearance and closing matches the task execution logic.

[0034] S204. Data Consistency and Expansion. During the integration phase, the system performs consistency verification to ensure that the ad-free and ad-supported versions maintain consistency in interface structure, number of controls, and semantic tags, with controlled differences only at ad nodes. The system's structural consistency verification algorithm compares the number of nodes, control mapping relationships, and semantic attributes; if differences are found, they are automatically corrected or flagged.

[0035] To improve the representativeness and scale of the dataset, the system employs an automated and intelligent expansion strategy. The system automatically identifies scalable scenarios based on task complexity, ad type distribution, and interface semantics, and generates multiple versions of samples through batch scripts. The system utilizes an interface analysis model to automatically determine the appropriate ad type for each page: splash screen or rewarded video ads are prioritized for launch or loading pages; native ads are prioritized for content display pages; banner ads are suitable for the bottom of the interface; and interstitial ads can be inserted on task interruption pages. The system assesses the task importance of each region based on saliency prediction and interaction heatmaps, automatically recommending ad positions and sizes in non-critical areas, achieving efficient and controllable data expansion. Through this mechanism, the dataset achieves automated expansion while maintaining consistency and authenticity.

[0036] S30. The final dataset is generated. The system archives the processing results of the static and dynamic groups to construct the final ad interference dataset. Using the task number as an index, the system establishes a unified directory structure. Each task directory contains both ad-free and ad-enabled samples, maintaining consistency in task objectives, interaction logic, and file structure. The system automatically generates a metadata index file (metadata.csv) recording information such as task number, application type, task steps, number of ads, ad type, and display location. It also generates a structure definition document (schema.md) to standardize field naming and constraints. The system checks the correspondence between screenshots, structure files, task description files, and event files through consistency verification to ensure the completeness and uniformity of task samples. After manual review, a bimodal ad interference dataset containing both static and dynamic groups is generated, providing high-quality foundational data for subsequent robustness evaluation.

[0037] Through the above data collection and processing flow, the system realizes the intelligent construction from ad-free task samples to ad-interference task samples, generating an evaluation dataset with unified format, structural standards and semantic alignment features, providing a standardized evaluation basis for different types of graphical user interface intelligent agents.

[0038] like Figure 2 As shown, this invention further provides a unified testing platform construction and evaluation analysis process. This platform, as the core supporting module of the advertising interference evaluation system, is used to achieve integrated operations such as task loading, agent scheduling, execution monitoring, multi-dimensional indicator calculation, and functional analysis. By performing performance analysis and comparison on different types of graphical user interface agents, a closed-loop process from advertising interference sample construction to agent performance evaluation is realized.

[0039] S40. Evaluation Phase. This phase includes three steps: building a unified testing platform, task execution and data recording, and indicator calculation and functional analysis. It is used to quantify the impact of advertising interference on the performance of graphical user interface agents and to provide a technical basis for algorithm improvement and functional optimization.

[0040] S401. Construction of a Unified Testing Platform. After the dataset is constructed, a testing platform is built based on a unified interface standard, implementing functional modules such as data loading, task scheduling, log collection, metric calculation, and visualization analysis. The platform adopts a modular design, mainly including: a task management module, a model scheduling module, a log collection module, a metric calculation module, and a visualization module. The task management module is responsible for parsing the task structure and advertising interference types; the model scheduling module is responsible for loading different types of graphical user interface agents or multimodal language models; the log collection module is responsible for capturing the execution status and user interface changes in real time; the metric calculation module automatically calculates performance metrics; and the visualization module generates performance comparison charts and task replay reports. The platform supports parallel testing and automated analysis, enabling unified evaluation of multiple models and multiple tasks simultaneously.

[0041] S402, Task Execution and Data Recording. After loading the task data, the unified testing platform calls the tested agent to execute the task operation. The system monitors the agent's operation trajectory, interface changes, and advertising interaction behavior in real time, and automatically records the execution log and intermediate state data. To ensure the reproducibility of the experiment, the platform provides runtime snapshots and task replay mechanisms, allowing the task to be executed repeatedly under the same input conditions to verify the model's robustness and decision stability.

[0042] S403. Multi-dimensional Indicator Calculation and Functional Analysis. Based on execution records, the platform calculates core indicators such as task completion rate, average execution time, error rate, ad detection accuracy, path recovery rate, and ad response latency. It also supports extended functional analysis, including task success path replay, ad trigger frequency statistics, misclick distribution analysis, recovery time sequence analysis, and strategy comparison and evaluation. The system automatically generates performance reports and visualization charts to compare performance differences under different agents, ad types, and interference intensities.

[0043] pass Figure 2 As shown in the diagram, the platform enables end-to-end evaluation from task loading and model execution to performance quantification. This platform not only possesses data-driven multi-dimensional indicator analysis capabilities but also supports flexible expansion with new functional modules, making it widely applicable for performance verification and robustness analysis of graphical user interface agents in advertising-interference scenarios.

[0044] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for automatically constructing an advertising interference evaluation benchmark for mobile graphical user interface intelligent agents, characterized in that, include: S1, based on the interface hierarchy tree parsing algorithm, abstracts and models the interface structure and control semantics of mobile applications, and collects and forms static group data and dynamic group data. S2 injects multiple types of advertising elements into the interface of static group data through template matching and generative advertising synthesis model; The actual ad display process is collected in the dynamic group data through the integrated advertising SDK; finally, static no-ad samples and paired static with-ad samples, dynamic no-ad samples and paired dynamic with-ad samples are obtained. S3 employs a multimodal detection and interface alignment algorithm to automatically identify the advertising area, advertising type, close control position, and advertising display duration in static and dynamic advertising samples, and outputs a unified structured annotation file, maintaining a correspondence with the task events of the sample on the timeline; S4, using the action sequence alignment module, inserts advertising processing actions into task events with static and dynamic ad samples, generating corresponding behavior event files; S5. The static group data and dynamic group data processed in steps S2 to S4 are used to construct an ad interference dataset. The task number, ad type and ad location distribution are recorded through a metadata index file to realize the mapping between task semantics and ad interference behavior. S6 generates evaluation metrics based on the advertising interference dataset, including task completion rate, misoperation rate, advertising detection accuracy, path recovery rate, and response latency, to quantify the impact of advertising interference on agent performance. S7 connects the aforementioned ad interference dataset to the testing platform through a unified interface to achieve robustness comparison and automated evaluation of different graphical user interface agents or multimodal language models.

2. The method for automatically constructing an advertising interference evaluation benchmark for mobile graphical user interface intelligent agents according to claim 1, characterized in that, In step S1, during the process of collecting and forming static group data, only file reading and metadata organization are performed, without modifying the task logic or interface content, to ensure the structural integrity and reproducibility of the data.

3. The method for automatically constructing an advertising interference evaluation benchmark for mobile graphical user interface intelligent agents according to claim 1, characterized in that, In step S2, the actual ad display process is collected in the dynamic group data by integrating the ad SDK. Specifically, this includes: embedding the ad SDK into the mobile application to generate an application with ads; running the ad-free and ad-ad applications on real or virtual devices respectively; fully capturing the entire process of ad loading, display, and closing; and saving the corresponding interface screenshots, interface structure files, and event logs.

4. The method for automatically constructing an advertising interference evaluation benchmark for mobile graphical user interface intelligent agents according to claim 1, characterized in that, In step S3, the multimodal detection and interface alignment algorithm adopts a combination of visual detection and XML hierarchical analysis. The Vision-XML Alignment module simultaneously identifies the ad area, ad type, close control position, and ad display duration, and outputs a unified structured annotation file.

5. The method for automatically constructing an advertising interference evaluation benchmark for mobile graphical user interface intelligent agents according to claim 1, characterized in that, In step S4, ad processing actions are inserted into both static and dynamic task events containing ad samples, specifically as follows: For splash screen ads, generate an ad processing action: detect the ad, wait for it to close, click to close, and then continue the task. For interstitial ads, generate an ad processing action that involves an operation chain of detecting the ad, clicking to close, and continuing the task. For banner ads, generate an ad processing action sequence of events: detect the ad, avoid the ad area, and continue the task. For incentivized video ads, generate an ad processing action that involves detecting the ad, playing the ad, waiting for the ad to end, and then continuing the task. For native ads, generate ad processing actions that identify ad content, adjust interactive focus, and continue the task.

6. The method for automatically constructing an advertising interference evaluation benchmark for mobile graphical user interface intelligent agents according to claim 1, characterized in that, In step S5, the ad interference dataset is indexed by the task number of the sample. Each task number contains both ad-free and ad-containing samples in its corresponding directory. The two samples are consistent in terms of task objectives, interaction logic, and file structure.

7. The method for automatically constructing an advertising interference evaluation benchmark for mobile graphical user interface intelligent agents according to claim 1, characterized in that, In step S5, during the construction of the ad interference dataset, samples that do not meet the alignment criteria are automatically filtered out by performing anomaly detection and resampling on the task execution logs and ad interference intensity.

8. The method for automatically constructing an advertising interference evaluation benchmark for mobile graphical user interface intelligent agents according to claim 1, characterized in that, In step S7, the testing platform includes a task execution interface and an evaluation metric calculation module; The task execution interface supports different graphical user interface agents or multimodal language models to automatically execute tasks in the same dataset environment; the evaluation index calculation module automatically generates task completion rate, misoperation rate, ad detection accuracy, path recovery rate and response latency based on the event logs and annotation files during the execution of sample task events, and outputs evaluation report files in a unified format.

Images