Efficient mobile advertisement detection system based on multi-modal proxy user interface navigation

Abstract

The application relates to the technical field of mobile application automatic test and security analysis, and discloses an efficient mobile advertisement detection system based on multi-modal agent user interface navigation, which comprises an offline portrait construction subsystem, a multi-modal inference guided UI navigation subsystem and a memory driven runtime optimization subsystem. The offline subsystem generates a prior knowledge base by combining static analysis and dynamic detection, the online navigation subsystem generates unified component representation data by using a UI unified perception module, solves the difference between cross-rendering technologies, and constructs a structured prompt word by combining the prior knowledge to drive a large language model to output an interactive instruction, and the runtime optimization subsystem retrieves a historical interaction summary by calculating the similarity of interface feature vectors, and stores the newly generated advertisement triggering experience into a vector database. The application realizes efficient and automatic detection of complex and non-standard interface advertisements, and significantly improves the detection coverage and execution efficiency.

Description

Technical Field

[0001] This invention relates to the field of automated testing and security analysis technology for mobile applications, specifically a high-efficiency mobile advertising detection system based on multimodal proxy user interface navigation. Background Technology

[0002] With the development of the mobile internet industry, in-app advertising has become one of the main ways for mobile application developers to generate revenue. However, in pursuit of higher commercial returns, some applications engage in illegal practices such as abusing pop-up ads, hiding ad closing links, or setting deceptive clickbait. This not only harms user experience but also poses security risks. Therefore, automated detection and compliance analysis of mobile application advertising behavior are particularly important.

[0003] Current automated mobile application detection technologies primarily rely on accessibility interfaces provided by the operating system or standard view hierarchy structures to obtain interface information. This approach performs adequately when handling conventional applications developed using native controls, but it has limitations when dealing with applications developed using game engines or non-standard UI frameworks. Many game applications or applications that directly draw their interfaces using graphics rendering libraries do not have their internal controls mounted on the standard system view tree, making it impossible for detection tools to obtain specific control coordinates or attributes. In such cases, existing detection systems often only identify a blank container node, failing to perceive buttons or text information within the interface, thus hindering effective monitoring of advertising violations in these applications.

[0004] Traditional dynamic detection methods typically employ random traversal strategies or script execution based on preset rules. Random traversal strategies lack an understanding of the application's internal logic, often resorting to a large number of disordered click attempts to cover the application page. This approach is inefficient when dealing with ad placements with complex triggering logic or deeply hidden ad locations. Simple dynamic detection fails to effectively utilize static code information within the application's installation package, resulting in a lack of clear target guidance during the detection process. It struggles to reach specific ad business code paths within a limited timeframe, leading to a high false negative rate.

[0005] Existing automated testing or detection solutions are typically executed independently for a single application. Once the detection system completes the analysis of an application, the successful paths or strategies accumulated during the exploration process are often discarded. When faced with large-scale application detection tasks, even if different applications integrate the same adware development kit or have similar ad triggering logic, the system still needs to explore and experiment again. This lack of experience storage and reuse mechanisms leads to the repeated consumption of computing resources in similar scenario reasoning, limiting the overall operational efficiency and response speed of the detection system in large-scale scenarios. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a high-efficiency mobile advertising detection system based on multimodal proxy user interface navigation. It solves the problems of being unable to identify non-standard rendered interfaces due to over-reliance on standard view interfaces, blind dynamic detection due to lack of static feature guidance leading to easy omission of complex advertising positions, and low efficiency and repeated resource consumption during large-scale detection due to the lack of experience reuse mechanism.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] The first aspect of this invention provides a high-efficiency mobile advertising detection system based on multimodal agent user interface navigation. The system mainly consists of an offline profile construction subsystem, a UI navigation subsystem guided by multimodal reasoning, and a memory-driven runtime optimization subsystem.

[0009] The offline profile building subsystem is configured to perform preprocessing before the mobile application runs, generating a prior knowledge base by combining static feature analysis and dynamic random probing techniques. This prior knowledge base includes screen prior data, slot prior data, trigger prior data, and network prior data. Specifically, the offline profile building subsystem generates screen prior data by parsing the target mobile application's manifest file to extract permission declarations and component registration information; it generates slot prior data by parsing layout resource files and matching view node class names with a pre-set ad development kit list to identify potential ad container locations; and it generates trigger prior data by analyzing the bytecode call graph and binding ad-related application interface call instructions back to specific Activity components, establishing an attribution mapping relationship from bottom-level calls to top-level components. Furthermore, this subsystem drives the target mobile application to perform random traversal tests, constructs a coarse-grained UI transition graph, and establishes a causal relationship between network request records and user interaction events within a preset time window based on a timestamp alignment algorithm. In this way, the system has basic structured data about the ad triggering location and logic at the beginning of the application startup, which solves the problem of low traversal efficiency due to lack of prior information.

[0010] The multimodal reasoning-guided UI navigation subsystem is configured to perform reasoning-based interaction decisions during the runtime of the target mobile application. This subsystem utilizes a unified UI awareness module to perform cross-rendering consistency processing, addressing the interface heterogeneity issues arising from different application development frameworks. For interfaces based on a standard view hierarchy, structured data of controls is extracted using accessibility service interfaces; for non-standard interfaces based on canvas rendering, object detection models are used to identify control area coordinates, and semantic descriptive text is generated in conjunction with a visual language model. All of this data is integrated into unified component representation data.

[0011] Building upon this foundation, the multimodal reasoning-guided UI navigation subsystem utilizes a multimodal reasoning agent module to construct structured prompts. These structured prompts aggregate heterogeneous information into a text sequence understandable by the large language model, specifically comprising four logical data blocks: a system instruction block defining model role parameters and task objectives; a current interface observation block containing serialized unified component representation data; a priori and policy context block containing prior knowledge base matching data and local UI transformation graph information; and a past experience block containing summaries of historical interaction paths. This subsystem inputs the structured prompts into the large language model, leverages the model's context-based generation capabilities to output interactive action commands and advertising relevance scores, and dynamically adjusts the priority of the exploration path based on these scores.

[0012] The memory-driven runtime optimization subsystem is configured to enable cross-application experience reuse and data accumulation updates. This subsystem utilizes a text embedding model to map the structural and semantic features of the interface into high-dimensional feature vectors. At runtime, the subsystem calculates the similarity between the feature vector of the current interface and the feature vectors stored in the historical experience database. This similarity calculation is achieved by performing a dot product operation on the two feature vectors and dividing by their modulo. When the similarity meets a preset condition, the system retrieves and reuses historical interaction path summaries to provide a reference for current exploration. When a successful ad trigger is detected, the subsystem extracts the UI state fingerprint of the current state and inputs the operation sequence that triggered the ad into a large language model to generate an interaction path summary. Subsequently, the feature-vectorized UI state fingerprint and the interaction path summary are associated and stored in a vector database. This mechanism enables the system to accumulate and utilize past successful experiences, improving detection efficiency for complex application scenarios and complex ad logic.

[0013] The second aspect of this invention provides an efficient mobile advertising detection method based on multimodal proxy user interface navigation. The method includes: utilizing an offline profiling subsystem to perform multi-layer static analysis and lightweight dynamic detection on a target mobile application, constructing a prior knowledge base containing multi-dimensional prior data; during the online detection phase, using a unified UI perception module to identify the rendering type of the current interface and generate unified component representation data that eliminates data format differences; utilizing a memory-driven runtime optimization subsystem to calculate the similarity between the current interface and historical advertising triggering scenarios in the semantic space, and retrieving historical interaction path summaries; constructing structured prompt words based on the prior knowledge base, unified component representation data, and historical interaction path summaries, and inputting these structured prompt words into a large language model to obtain interaction decisions and relevance scores; after driving the mobile terminal to execute interaction commands and successfully triggering an advertisement, extracting the state fingerprint and operation sequence summary of the current interface, storing them as new experience in a historical experience base for reuse in subsequent detection processes.

[0014] A third aspect of the present invention provides a non-transient computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the above-described efficient mobile advertising detection method based on multimodal proxy user interface navigation.

[0015] The technical solution of this invention achieves automated detection of mobile application advertisements by integrating offline prior knowledge, online multimodal perception reasoning, and a vector retrieval-based memory mechanism. It is specifically optimized for game engine rendering interfaces and complex advertisement triggering logic. Through the combination of structured prompts and a large language model, it can complete advertisement triggering tasks through semantic understanding even in the absence of explicit rule guidance. Furthermore, through an experience reuse mechanism, detection efficiency gradually improves as the number of detection tasks increases.

[0016] This invention provides a high-efficiency mobile advertising detection system based on multimodal proxy user interface navigation. It has the following beneficial effects: 1. This invention achieves a unified representation of standard view hierarchy interfaces and canvas rendering interfaces by combining a unified UI perception module with accessibility service interfaces and hybrid visual detection technology. For applications developed using game engines or non-native UI frameworks, the system can extract the coordinates and semantic descriptions of controls using object detection models and visual language models, and transform them into unified component representation data. This mechanism solves the problem that existing detection tools, which rely solely on the system control tree, cannot effectively identify ad trigger points in non-standard interfaces, ensuring the system's detection coverage across rendering technology scenarios.

[0017] 2. This invention utilizes an offline profiling subsystem to generate a prior knowledge base containing screen, slot, trigger, and network features, and integrates this prior knowledge base into the structured prompts of online navigation. This strategy of fusing static code analysis results with lightweight dynamic detection data provides clear navigation goals and spatial constraints for large language models. Compared to purely random traversal or reinforcement learning methods without prior knowledge, this design effectively reduces invalid interaction attempts and improves the positioning accuracy and detection success rate for hidden ad triggering logic.

[0018] 3. This invention establishes an experience reuse mechanism based on vector retrieval through a memory-driven runtime optimization subsystem. The system abstracts historically successful ad-triggered operation sequences into experience data entries containing interface state fingerprints and operation summaries, and stores them in a vector database. When detecting new applications, the system can quickly retrieve and reuse previously effective strategies based on the similarity of interface semantics. This reduces the redundant consumption of inference resources on large language models and significantly improves execution efficiency and system response speed in large-scale application detection scenarios. Attached Figure Description

[0019] Figure 1This is a system architecture diagram of the present invention; Figure 2 A schematic diagram of the system flow of this invention; Figure 3 Structural diagram of the multimodal mobile advertising detection system of the present invention; Figure 4 Prior percentage diagram of different categories in this invention; Figure 5 The percentage of different categories of advertisements in this invention; Figure 6 Diagram of the cross-rendering consistency scheme of the present invention; Figure 7 Inference example diagrams of the system of the present invention in two types of applications; Figure 8 The cross-application experience summary and reuse example diagram of the present invention; Figure 9 Example diagram of the structured prompt words of the present invention; Figure 10 A schematic diagram illustrating representative successful cases of this invention; Figure 11 Overall detection performance diagram of the present invention; Figure 12 The detection performance charts of this invention for different advertising categories; Figure 13 Overall ablation experimental performance diagram of the present invention; Figure 14 The ablation test performance diagram of the present invention for different advertising categories; Figure 15 The ablation performance diagram of this invention for visual cues; Figure 16 Experimental performance graphs of different large language models of the present invention; Figure 17 Distribution of token consumption for different large language models in this invention; Figure 18 The detection performance graphs of different mobile phones according to the present invention; Figure 19 Example diagram of the system for detecting malicious advertisements according to the present invention; Figure 20 An example diagram illustrating the extension of this invention to abuse of permissions; Figure 21 A schematic diagram illustrating a representative failure case of this invention. Detailed Implementation

[0020] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please see the appendix Figure 1 -Appendix Figure 21 This invention provides a high-efficiency mobile advertising detection system based on multimodal proxy user interface navigation, comprising: an offline profile construction subsystem 100, a multimodal inference-guided UI navigation subsystem 200, and a memory-driven runtime optimization subsystem 300. The offline profile construction subsystem 100, the multimodal inference-guided UI navigation subsystem 200, and the memory-driven runtime optimization subsystem 300 are connected via a data transmission bus for data interaction.

[0022] The offline profile building subsystem 100 is configured to preprocess the target mobile application before formal detection, generating a prior knowledge base through a combination of static feature extraction and lightweight dynamic detection. Specifically, the offline profile building subsystem 100 includes a static analysis unit 110 and a dynamic detection unit 120.

[0023] The static analysis unit 110 is configured to decompile and parse the installation package of the mobile application. The static analysis unit 110 includes a screen prior extraction module 111, a slot prior extraction module 112, and a trigger prior extraction module 113.

[0024] The screen prior extraction module 111 is configured to parse the manifest file of the mobile application, extract the permission declaration list, component registration information and metadata, and identify the application-level advertising integration signals based on the permission declaration list and component registration information.

[0025] The slot prior extraction module 112 is configured to parse the layout resource file of the mobile application, identify the ad container control by matching the package name in the layout file with the pre-stored ad development kit prefix list, and record the resource identifier of the ad container control and the coordinate position of the ad container control in the view hierarchy.

[0026] The trigger prior extraction module 113 is configured to perform control flow analysis on the bytecode of the mobile application, construct a call relationship graph with the Activity component as the root node, identify advertising-related application interface calls and lifecycle callback functions, and bind advertising-related application interface calls back to specific Activity components through the call relationship graph to generate trigger prior data.

[0027] The dynamic detection unit 120 is configured to perform random automated tests for a preset duration. It uses a pseudo-random strategy to drive the mobile application to run and sample to generate a coarse-grained UI transition graph. The UI transition graph contains nodes representing Activity states and edges representing triggering events. The dynamic detection unit 120 also collects runtime system logs in conjunction with the log capture module. It extracts advertising network request events by matching advertising domain names and feature parameters, and uses timestamp comparison to associate user interaction events with network request events occurring within a preset time window to generate network prior data.

[0028] The multimodal reasoning-guided UI navigation subsystem 200 is configured to perceive the interface state and plan interaction actions in real time during mobile application runtime. The multimodal reasoning-guided UI navigation subsystem 200 includes a UI unified perception module 210, a context state management module 220, and a multimodal reasoning agent module 230.

[0029] The UI Unified Awareness Module 210 is configured to handle the heterogeneity issues between application interfaces based on standard views and those based on canvas rendering. When a standard view is detected, the UI Unified Awareness Module 210 calls the accessibility service interface to obtain the class name, resource identifier, and text label of the control. When a canvas-rendered view is detected, the UI Unified Awareness Module 210 calls a hybrid vision detector. The hybrid vision detector uses an object detection model to extract the control area and uses geometric heuristics rules for correction. Subsequently, it calls a visual language model to generate a semantic description of the control area. The UI Unified Awareness Module 210 integrates the above information and outputs it as component representation data in a unified format.

[0030] The context state management module 220 is configured to maintain the current exploration state, which includes a short-term interaction history buffer and an online dynamically updated UI transition graph. The context state management module 220 writes the most recently executed event sequence and the results of ad triggering into the short-term interaction history buffer.

[0031] The multimodal reasoning agent module 230 is built based on a large language model and configured to receive structured prompts. The structured prompts are composed of static prior data generated by the static analysis unit 110, component representation data in a unified format generated by the UI unified awareness module 210, exploration state data provided by the context state management module 220, and historical experience data retrieved by the memory-driven runtime optimization subsystem 300. The multimodal reasoning agent module 230 inputs the structured prompts into the large language model and parses the output of the large language model to obtain the next interaction action decision and the advertising relevance score of the current state.

[0032] The memory-driven runtime optimization subsystem 300 is configured to reuse successful ad triggering strategies across applications. The memory-driven runtime optimization subsystem 300 includes an experience extraction module 310, a vector storage module 320, and an experience retrieval module 330.

[0033] The experience extraction module 310 is configured to extract the UI state fingerprint of the state before the trigger time and the interaction path summary generated by the large language model when the successful triggering of the advertisement is detected. The UI state fingerprint includes the set of class names of controls in the interface, the set of resource identifiers and the set of text content.

[0034] The vector storage module 320 is configured to use a text embedding model to map UI state fingerprints into feature vectors, and associate the feature vectors with the corresponding interaction path summaries and store them in the vector database.

[0035] The experience retrieval module 330 is configured to calculate the similarity between the feature vector of the current UI state and the feature vector of historical experiences stored in the vector database at runtime. The experience retrieval module 330 uses a cosine similarity algorithm, and the calculation formula is as follows:

[0036] in, The feature vector representing the currently detected UI state. This represents the feature vector of the UI state fingerprint stored in the historical experience database. Represents the magnitude of a vector. express and The similarity score between the two is calculated. When the similarity score exceeds a preset threshold, the experience retrieval module 330 retrieves the corresponding interaction path summary from the vector database and sends the interaction path summary to the multimodal reasoning agent module 230. All the above subsystems and modules are deployed in a general-purpose computing device, which connects to the mobile terminal device running the target mobile application via a debug bridge interface to achieve command control and data acquisition.

[0037] This invention provides a mobile advertising detection method based on multimodal proxy user interface navigation, executed by a mobile advertising detection system. The mobile advertising detection method includes steps S100 to S600: In step S100, the offline profile construction subsystem 100 performs offline analysis on the target mobile application to generate a prior knowledge base containing screen prior data, slot prior data, trigger prior data, and network prior data.

[0038] In step S100, the static analysis unit 110 decompiles the mobile application's installation package. The screen prior extraction module 111 extracts metadata from the manifest file. The slot prior extraction module 112 scans the layout resource file to locate potential ad container controls. The trigger prior extraction module 113 analyzes the bytecode call relationship graph and establishes an attribution mapping relationship from the underlying application interface calls to the top-level Activity components. Simultaneously, the dynamic detection unit 120 drives the mobile application to perform a random traversal for a preset duration to collect the initial UI transition graph and uses the log capture module to extract ad network request features that are temporally correlated with user operations to form network prior data.

[0039] In step S200, the multimodal reasoning-guided UI navigation subsystem 200 loads the prior knowledge base and starts the online detection loop for the mobile application.

[0040] In step S300, the UI unified awareness module 210 obtains the unified component representation data of the current interface.

[0041] In step S300, the UI Unified Awareness Module 210 first detects the class name or view hierarchy depth of the current interface's root node to determine the rendering type of the current interface. If the current interface is a standard view, the UI Unified Awareness Module 210 calls the accessibility service interface to read the control tree information; if the current interface is a canvas rendering view, the UI Unified Awareness Module 210 activates the hybrid vision detector, identifies control areas through the object detection model, and generates semantic descriptions by combining them with the visual language model. The UI Unified Awareness Module 210 integrates the acquired control class names, resource identifiers, text labels, or semantic descriptions into component representation data in a unified format.

[0042] In step S400, the memory-driven runtime optimization subsystem 300 retrieves historical experience based on the unified component representation data.

[0043] In step S400, the experience retrieval module 330 extracts the UI state fingerprint of the current interface and calculates the similarity score between the feature vector of the UI state fingerprint and the historical experience feature vector stored in the vector database. When the similarity score is higher than a preset threshold, the experience retrieval module 330 outputs the corresponding historical interaction path summary; if the similarity score does not exceed the preset threshold, no data is returned.

[0044] In step S500, the multimodal reasoning agent module 230 generates and executes interactive decisions.

[0045] In step S500, the multimodal reasoning agent module 230 constructs structured prompts, which include: static prior data from the offline profile construction subsystem 100, unified component representation data from the UI unified perception module 210, the current exploration state (including local UI transformation graphs and short-term interaction history sequences) from the context state management module 220, and the historical interaction path summary retrieved in step S400.

[0046] The multimodal inference agent module 230 inputs structured prompts into a pre-trained large language model. The multimodal inference agent module 230 parses the text output of the large language model to extract the next operation instruction for the current interface and the advertising relevance score for the current state. The mobile advertising detection system drives the mobile terminal to perform corresponding actions based on the operation instructions.

[0047] In step S600, the context state management module 220 updates the context state and processes the ad triggering result.

[0048] In step S600, the context state management module 220 updates the executed operations and interface jump results to the short-term interaction history buffer and UI transformation map, and updates the running estimate of the current exploration path according to the advertising relevance score.

[0049] The mobile advertising detection system monitors interface changes and system network logs in real time, and determines whether an advertisement has been successfully triggered by matching preset advertising campaign features or advertising network request features. If an advertisement trigger is detected, the experience extraction module 310 extracts the UI state fingerprint of the state before the advertisement trigger and the interaction path summary summarized by the large language model. The vector storage module 320 vectorizes the UI state fingerprint and associates the feature vector of the UI state fingerprint with the interaction path summary and stores it in the vector database.

[0050] The mobile advertising detection system repeats steps S300 to S600 until the preset detection duration or detection step limit is reached.

[0051] The offline profile construction subsystem 100 performs multi-layer static prior knowledge extraction through the static analysis unit 110. The specific process is as follows: When the static analysis unit 110 is running, the screen prior extraction module 111 reads and parses the manifest file of the target mobile application. The screen prior extraction module 111 traverses the permission declaration nodes in the manifest file, identifying whether the manifest file contains network access permissions (e.g., android.permission.INTERNET) and specific permissions required by the adware development kit. Simultaneously, the screen prior extraction module 111 scans the application component registration list, searching for Activity, Service, and Receiver components whose names contain preset advertising keywords (e.g., "AdActivity," "AdService"). The screen prior extraction module 111 also extracts metadata tags from the manifest file, verifying whether the manifest file contains configuration parameters required for ad platform initialization. The screen prior extraction module 111 integrates the extracted permission declaration nodes, application component registration list, and metadata tags into screen prior data, which is used to indicate whether the mobile application integrates advertising functionality and potential ad component entry points.

[0052] The slot prior extraction module 112 scans the mobile application's resource directory to locate all layout resource files. It then parses the XML structure tree of the layout resource files, traversing the view nodes within the tree. The module obtains the class name attribute of each view node and performs prefix matching with a pre-built ad software development kit (SDK) class name library. Upon successful matching, the module determines the view node to be a potential ad container control. It records the resource identifier of the ad container control, its depth position in the layout hierarchy, and its parent container type. Furthermore, the module infers the ad type based on the ad container control's layout attributes (including width and height) and control naming rules to distinguish between banner ad slots and interstitial ad slots. Finally, the module outputs slot prior data containing the ad container's position and type, providing structured spatial guidance for subsequent UI navigation.

[0053] The trigger prior extraction module 113 converts the executable file of the mobile application into an intermediate code representation and constructs a control flow graph and a function call graph based on the intermediate code representation. The trigger prior extraction module 113 searches the function call graph for target call instructions that match the signature of the preset advertising application interface. The target call instructions include advertising loading instructions (e.g., loadAd), advertising display instructions (e.g., show), and advertising initialization instructions (e.g., init).

[0054] The trigger prior extraction module 113 performs reverse control flow analysis, starting from the identified target call instruction and tracing back along the function call chain until it reaches the top-level Activity component lifecycle callback function or user interaction event listener function. Through the reverse control flow analysis process, the trigger prior extraction module 113 establishes an attribution mapping relationship from the underlying advertising application interface call to the specific Activity component.

[0055] The trigger prior extraction module 113 prioritizes triggers based on the semantic type of the target invocation instruction. Specifically, the trigger prior extraction module 113 sets triggers containing ad display instructions as the first priority, triggers containing ad loading instructions as the second priority, and triggers containing ad initialization instructions as the third priority. The trigger prior extraction module 113 outputs trigger prior data containing attribution mapping relationships and priority information. This trigger prior data is used to predict the timing and logic of ad triggering at runtime.

[0056] The static analysis unit 110 aggregates and stores the screen prior data generated by the screen prior extraction module 111, the slot prior data generated by the slot prior extraction module 112, and the trigger prior data generated by the trigger prior extraction module 113 into the prior knowledge base. The prior knowledge base is called by the UI navigation subsystem 200 guided by multimodal reasoning.

[0057] The offline profile construction subsystem 100 performs lightweight dynamic detection and network prior extraction through the dynamic detection unit 120. The specific process is as follows: The dynamic detection unit 120 is configured to drive the target mobile application to perform a random traversal test for a preset duration in a controlled operating environment. During the random traversal test, the dynamic detection unit 120 generates a sequence of input events using a pseudo-random strategy and sends the sequence of input events to the user interface of the mobile application. The sequence of input events includes click coordinate events, swipe trajectory events, and system key events. The dynamic detection unit 120 monitors the interface response and lifecycle changes of the Activity components of the mobile application in real time and records the interface state transition data triggered by each input event. Based on the interface state transition data, the dynamic detection unit 120 constructs an initial coarse-grained UI transition graph, which consists of nodes representing interface states and directed edges representing input events.

[0058] While performing random traversal tests, the dynamic detection unit 120 calls the system log interface to capture the runtime log stream of the mobile application. The dynamic detection unit 120 loads a pre-built advertising network feature library, which contains a list of known advertising service provider domain names and key-value pairs of specific advertising parameters from the Uniform Resource Locator (URL). The dynamic detection unit 120 uses the advertising network feature library to perform real-time filtering and matching on the runtime log stream. When the dynamic detection unit 120 detects a network request record in the runtime log stream that matches the advertising network feature library, it extracts the generation timestamp of the network request record.

[0059] The dynamic detection unit 120 performs time-series alignment analysis to generate network prior data. Based on the generation timestamp of the network request record, the dynamic detection unit 120 backtracks and retrieves user interaction events within a preset time window from the recorded input event sequence. When a target user interaction event is retrieved, the dynamic detection unit 120 establishes a causal relationship mapping between the target user interaction event and the network request record. The dynamic detection unit 120 marks this causal relationship mapping as network prior data, which indicates the specific interface nodes and interaction types that can trigger background ad loading traffic in the coarse-grained UI transition graph.

[0060] The offline profile construction subsystem 100 stores the coarse-grained UI transformation graph and network prior data generated by the dynamic detection unit 120 into a prior knowledge base. The prior knowledge base uses the Activity component class name or interface structure hash value as the index key to align the coarse-grained UI transformation graph and network prior data with the static prior data generated by the static analysis unit 110.

[0061] The multimodal reasoning-guided UI navigation subsystem 200 performs cross-rendering consistent UI representation processing through the UI unified awareness module 210, as follows: The UI Unified Awareness Module 210 obtains the root node of the view hierarchy of the current interface of the mobile application. The UI Unified Awareness Module 210 reads the class name attribute and the number of child nodes of the root node. If the class name of the root node belongs to a native Android view class (including LinearLayout, FrameLayout, and RelativeLayout) and the root node contains a traversable child node tree, the UI Unified Awareness Module 210 determines that the current interface is a standard user interface based on hierarchy. If the class name of the root node belongs to a graphics rendering container class (including SurfaceView, GLSurfaceView, and UnityPlayer) or the view hierarchy depth of the root node is lower than a preset depth threshold, the UI Unified Awareness Module 210 determines that the current interface is a non-standard user interface based on canvas rendering.

[0062] When the current interface is determined to be a hierarchical standard user interface, the UI Unified Awareness Module 210 calls the accessibility service interface to traverse each leaf node in the view tree. The UI Unified Awareness Module 210 extracts the metadata of the leaf nodes, which includes control class name, resource identifier, screen coordinate bounding box, text content, and content description attributes. The UI Unified Awareness Module 210 uses visibility attributes to filter out invisible nodes and maps the metadata of the remaining leaf nodes to the attribute fields of the unified component object.

[0063] When the current interface is determined to be a non-standard user interface based on canvas rendering, the UI unified perception module 210 initiates a hybrid vision detection process. The UI unified perception module 210 captures a screenshot of the current interface and inputs the screenshot into the hybrid vision detector. The hybrid vision detector performs deep learning object detection and heuristic visual analysis in parallel.

[0064] In deep learning object detection, the hybrid vision detector uses a pre-trained object detection model to infer from screenshots. The object detection model identifies control categories specific to game or advertising scenarios, including close buttons, skip buttons, download buttons, and reward claim icons. The object detection model outputs the bounding box coordinates and confidence scores for each identified control category.

[0065] In the heuristic visual analysis process, a hybrid visual detector uses an edge detection algorithm to scan the screenshot to extract closed contours. The hybrid visual detector filters out closed contours located in the four corners of the screen and rectangular regions with high-contrast edges, and uses an aspect ratio threshold to filter out non-control shapes, marking the remaining areas as general interaction candidate regions. This heuristic visual analysis process is used to recall atypical controls that the object detection model failed to identify.

[0066] The UI unified perception module 210 performs a region merging operation, calculating the intersection-over-union (IoU) ratio between the bounding boxes output by the object detection model and the general interactive candidate regions output by the hybrid vision detector. If the IoU ratio is greater than a preset overlap threshold, the UI unified perception module 210 retains the region with the higher confidence score; if the IoU ratio is less than or equal to the preset overlap threshold, the UI unified perception module 210 retains both regions, thereby generating a candidate control region list.

[0067] The UI unified perception module 210 performs image cropping on each candidate control area and inputs the cropped image fragments into the visual language model. The visual language model generates natural language text describing the visual content or functional intent of the image fragments.

[0068] The UI Unified Awareness Module 210 uses the bounding box coordinates of the candidate control region as the position attribute and the category output by the object detection model or the natural language text generated by the visual language model as the semantic attribute to construct a unified component object. The UI Unified Awareness Module 210 outputs a list of current interface descriptions containing all unified component objects.

[0069] The multimodal inference agent module 230 is configured to quantitatively evaluate the advertising relevance of the current state. The multimodal inference agent module 230 extracts a local subgraph structure centered on the current node from the UI transition graph. This local subgraph structure includes the current node and its adjacent nodes within a preset depth range. The multimodal inference agent module 230 integrates the local subgraph structure, the content of the short-term interaction history buffer, and the stagnant state markers into structured prompts. The multimodal inference agent module 230 inputs the structured prompts into a large language model and parses the output of the large language model to obtain an immediate relevance score between the current interface state and the advertising trigger target. The immediate relevance score is a normalized numerical value; a higher value indicates a closer logical distance between the current state and the potential advertising trigger point.

[0070] The multimodal inference agent module 230 maintains a runtime estimate using an exponential moving average algorithm. The multimodal inference agent module 230 calculates the runtime estimate for the current time step according to the following formula:

[0071] in, Indicates the current time step The estimated running value, Indicates the previous time step The estimated running value, This indicates that the large language model is for the current time step. The output is an instant relevance score. This represents the preset smoothing factor. The multimodal inference agent module 230 will calculate the smoothing factor. Feedback is sent to the context state management module 220 to update the priority weight of the current node in the UI transition graph. When the threshold for abandonment is lower than the preset threshold, the multimodal reasoning agent module 230 generates a backtracking instruction or a restart instruction to drive the mobile application to exit the current exploration path.

[0072] The multimodal reasoning-guided UI navigation subsystem 200 performs the construction and reasoning of structured prompt words through the multimodal reasoning agent module 230. The specific process is as follows: The multimodal reasoning agent module 230 is configured to aggregate input data into structured prompts that conform to the input specifications of a large language model. The structured prompts are composed of four logical data blocks: system instruction block, current interface observation block, prior and policy context block, and past experience block.

[0073] The multimodal inference agent module 230 first generates a system instruction block. This block contains role definition parameters, task objective descriptions, decision constraint rules, and output format definitions. The role definition parameters configure the large language model as an Android automated testing agent. The task objective description instructs the large language model to detect and trigger hidden ads. The decision constraint rules restrict the large language model from generating duplicate click instructions or invalid area click instructions. The output format definition constrains the large language model to return decision results in a predefined machine-readable format (e.g., JavaScript object notation JSON). The data structure of the decision results includes an action type field, an operation object coordinate field or an identifier field, and an ad relevance score field for the current state.

[0074] The multimodal inference agent module 230 obtains the unified component representation data of the current interface from the UI unified perception module 210, and uses a text serialization algorithm to convert the unified component representation data into a current interface observation block. The current interface observation block enumerates the attribute information of all interactive components in the current interface in the form of a text list. The attribute information includes the component's index number, class name, resource identifier, screen coordinate range, text content, and semantic description generated by the visual language model. For canvas-based rendering interfaces, the multimodal inference agent module 230 directly writes the natural language description of the clipping region by the visual language model into the component description field corresponding to the current interface observation block.

[0075] The multimodal reasoning agent module 230 constructs a priori and policy context block by combining the prior knowledge base generated by the offline profiling subsystem 100. The multimodal reasoning agent module 230 traverses the component information of the current interface and matches it with static prior data. When the resource identifier or package name of the current component matches the slot prior data, the multimodal reasoning agent module 230 adds a "potential advertising container" tag to the current component in the prior and policy context block. When the current Activity component matches the trigger prior data, the multimodal reasoning agent module 230 writes high-priority exploration hint text into the prior and policy context block. Furthermore, the multimodal reasoning agent module 230 extracts local UI transition graph data and short-term interaction history sequences from the context state management module 220, and serializes the local UI transition graph data (including the list of visited adjacent nodes and the number of edge traversals) before writing it into the prior and policy context block.

[0076] The multimodal reasoning agent module 230 receives historical interaction path summaries from the memory-driven runtime optimization subsystem 300 and populates these summaries into the past experience block. The past experience block contains descriptions of operation sequences from historical successful cases similar to the current interface state. If the memory-driven runtime optimization subsystem 300 does not retrieve any experience exceeding a similarity threshold, the multimodal reasoning agent module 230 sets the past experience block to an empty string or fills it with preset default exploration strategy text.

[0077] The multimodal reasoning agent module 230 inputs the constructed structured prompts into the pre-trained large language model. The multimodal reasoning agent module 230 calls the reasoning interface of the large language model, which generates a text response based on the visual semantic information in the current interface observation block, the structured evidence in the prior and policy context blocks, and the heuristic knowledge in the past experience blocks.

[0078] The multimodal inference agent module 230 receives and parses the text output stream of the large language model. It uses regular expression matching or a JSON validator to extract the next interaction action instruction and the immediate relevance score from the text output stream. The interaction action instruction specifies the specific operation to be performed on the mobile terminal, including clicking, long-pressing, swiping, inputting text, or system-level back. The multimodal inference agent module 230 verifies the validity of the interaction action instruction and sends the verified instruction to the mobile terminal for execution. Simultaneously, the multimodal inference agent module 230 feeds back the extracted immediate relevance score to the context state management module 220 to update the path evaluation state.

[0079] The memory-driven runtime optimization subsystem 300 performs the definition and extraction processing of experience data through the experience extraction module 310. The specific process is as follows: The experience extraction module 310 is configured to construct an experience data structure. This data structure contains two core fields: a trigger state fingerprint field and an interaction path summary field. The trigger state fingerprint field uniquely identifies the interface state characteristics just before the ad is triggered. Its data content includes a set of class names, resource identifiers, and text content for all key controls in the interface. The interaction path summary field stores a natural language description of the valid operation sequence that led to the ad trigger. Its data content is a semantic summary of the original operation log.

[0080] The experience extraction module 310 monitors the operation of the mobile application. When the mobile advertising detection system determines that an advertisement has been successfully triggered based on network log matching results or interface state change results, the experience extraction module 310 executes the extraction process. The experience extraction module 310 first determines the preceding interface state at the time the advertisement triggering event occurs. The experience extraction module 310 obtains the unified component representation data corresponding to the preceding interface state from the UI unified perception module 210.

[0081] The experience extraction module 310 performs feature extraction operations on the unified component representation data to generate a trigger state fingerprint. The experience extraction module 310 traverses the component list, extracting the class name attribute, resource identifier attribute, and text label attribute for each interactive component. To ensure the uniqueness and determinism of the trigger state fingerprint, the experience extraction module 310 arranges the extracted attribute data according to a top-down, left-to-right sorting rule based on screen coordinates, and concatenates the arranged attribute data into a string sequence. The experience extraction module 310 removes dynamic layout parameters unrelated to the ad triggering logic from the string sequence. These dynamic layout parameters include specific pixel coordinate values, color attribute values, and control size values, thus forming the trigger state fingerprint.

[0082] The experience extraction module 310 obtains the recent operation record sequence from the short-term interaction history buffer of the context state management module 220. The recent operation record sequence contains all user interaction events executed within a preset time window before the advertisement is triggered. The experience extraction module 310 constructs a summary generation prompt word, which includes the recent operation record sequence, the corresponding interface jump result, and the causal inference instruction. The causal inference instruction instructs the large language model to identify and extract the key action steps and key waiting durations in the recent operation record sequence that led to the advertisement triggering, and ignore invalid random click operations.

[0083] The experience extraction module 310 inputs the summary-generated prompt words into the large language model. The experience extraction module 310 receives the natural language text output by the large language model and uses the natural language text as an interaction path summary.

[0084] The experience extraction module 310 associates and combines the generated trigger state fingerprint with the interaction path summary to construct an experience data entry. The experience extraction module 310 then sends the experience data entry to the vector storage module 320 for subsequent vectorization and persistent storage. Through this process, the experience extraction module 310 transforms the runtime operation log into reusable experience knowledge.

[0085] The memory-driven runtime optimization subsystem 300 performs the embedding and storage processing of experience data through the vector storage module 320, as follows: Vector storage module 320 is configured to receive experience data entries constructed by experience extraction module 310. Each experience data entry contains a trigger state fingerprint and an interaction path summary. Vector storage module 320 loads a pre-trained text embedding model configured to map natural language text or structured text sequences to a high-dimensional semantic space.

[0086] The vector storage module 320 transmits the serialized text form of the trigger state fingerprint as input data to the text embedding model. The text embedding model performs forward inference computation on the trigger state fingerprint and outputs a fixed-length numerical vector, i.e., the feature vector. The feature vector encodes the structural features, control semantic features, and layout topology features of the interface preceding the advertisement trigger time in the numerical space.

[0087] The vector storage module 320 establishes a connection with the vector database. The vector database is configured to support index construction and similarity retrieval of high-dimensional vectors. The vector storage module 320 performs L2 norm normalization on the feature vectors to adapt to the cosine similarity calculation standard.

[0088] The vector storage module 320 constructs storage records in the vector database. Each storage record contains feature vectors, the original text of the trigger state fingerprint, and a summary of the interaction path. The vector storage module 320 uses a hierarchical navigable small-world (HNSW) algorithm or an inverted file index (IVF) algorithm to construct a vector index for the feature vectors and persistently writes the storage records into the vector database.

[0089] The memory-driven runtime optimization subsystem 300 performs similarity-based retrieval and reuse processing through the experience retrieval module 330, as follows: The experience retrieval module 330 is configured to respond to interface state change events during the operation of the mobile application. When the UI unified awareness module 210 outputs the unified component representation data of the current interface, the experience retrieval module 330 performs a retrieval operation. The experience retrieval module 330 performs feature extraction and serialization processing on the unified component representation data. The experience retrieval module 330 extracts the class names, resource identifiers, and text content of all visible controls in the current interface, and uses the same preset spatial sorting rules and delimiters as the experience extraction module 310 to concatenate the extracted attribute data into a current state text sequence.

[0090] The experience retrieval module 330 calls the text embedding model in the vector storage module 320 to vectorize the current state text sequence. The text embedding model outputs a fixed-length numerical vector, i.e., the current feature vector (denoted as ). The experience retrieval module 330 accesses the vector database managed by the vector storage module 320, and performs an approximate nearest neighbor search using the current feature vector as the query object and a hierarchical navigation small-world index or an inverted file index.

[0091] The experience retrieval module 330 calculates the current feature vector and the historical experience feature vectors retrieved from the vector database (denoted as...). The cosine similarity between the current feature vector and the historical experience feature vector is calculated. The experience retrieval module 330 calculates the dot product of the current feature vector and the historical experience feature vector, and calculates the product of the magnitude of the current feature vector and the magnitude of the historical experience feature vector. The experience retrieval module 330 divides the dot product by the product of the magnitudes to obtain a normalized similarity score. The similarity calculation process quantifies the degree of closeness between the current interface structure and the historical ad triggering scenario in the semantic space.

[0092] The experience retrieval module 330 compares the calculated similarity score with a preset similarity matching threshold. The experience retrieval module 330 performs a filtering operation, eliminating candidate records with similarity scores lower than the similarity matching threshold. For candidate records with similarity scores higher than or equal to the similarity matching threshold, the experience retrieval module 330 sorts them in descending order of similarity score and selects the record ranked first as the matching experience.

[0093] The experience retrieval module 330 extracts associated interaction path summaries from the vector database based on selected matching experience. These interaction path summaries contain descriptions of the action steps that led to the triggering of historical ads. The experience retrieval module 330 sends the extracted interaction path summaries to the multimodal inference agent module 230. The multimodal inference agent module 230 writes the interaction path summaries into the past experience blocks of structured prompt words, serving as a reference for the large language model to generate the current interaction decision. If the experience retrieval module 330 does not find any record in the vector database with a similarity score higher than the similarity matching threshold, it returns an empty result to the multimodal inference agent module 230, which then adopts the default exploration strategy. This invention provides an electronic device 400, which is used to execute the mobile advertising detection method based on multimodal proxy user interface navigation described in the above embodiments.

[0094] Electronic device 400 includes bus 410, processor 420, communication interface 430, input device 440, output device 450, and memory 460. Bus 410 interconnects processor 420, communication interface 430, input device 440, output device 450, and memory 460 to realize signal transmission.

[0095] Processor 420 includes one or more central processing units (CPUs), graphics processing units (GPUs), or application-specific integrated circuits (ASICs). Processor 420 is used to read and execute computer program instructions stored in memory 460. By executing computer program instructions, processor 420 logically constructs and runs offline portrait construction subsystem 100, multimodal reasoning-guided UI navigation subsystem 200, and memory-driven runtime optimization subsystem 300.

[0096] The memory 460 includes a non-transient computer-readable storage medium. The memory 460 is used to store an operating system, application code, data structures, and a database. In this embodiment, the memory 460 explicitly stores program modules for performing a mobile advertising detection method, as well as a priori knowledge base, UI transition graph, short-term interaction history buffer, and vector database generated during system operation.

[0097] Communication interface 430 is used to establish a data transmission channel between electronic device 400 and an external target mobile terminal or network server. Communication interface 430 includes a wired transmission interface (e.g., a Universal Serial Bus (USB) interface, an Ethernet interface) or a wireless transmission interface (e.g., a Wireless Local Area Network (WiFi) interface). Communication interface 430 integrates control logic supporting Android Debug Bridge (ADB) protocol communication, used to send user interaction commands (including click, swipe, and key commands) to the connected mobile terminal and receive interface hierarchy description data, screenshot data streams, and system log data streams returned by the mobile terminal.

[0098] Input device 440 is used to receive external operation commands or parameter configurations, and includes a keyboard, mouse, touch screen, or microphone. Output device 450 is used to display test results, operation logs, and system status to the user, and includes a monitor, printer, or LED indicator lights.

[0099] The processor 420 performs the following operations by calling instructions from memory 460: controlling the offline profile building subsystem 100 to perform static analysis and dynamic detection on the mobile application to generate prior knowledge; controlling the UI unified perception module 210 to acquire and unify interface data of different rendering types; controlling the multimodal reasoning agent module 230 to generate interactive decisions based on structured prompts and a large language model; and controlling the memory-driven runtime optimization subsystem 300 to store and retrieve advertising trigger experience data.

[0100] This invention also provides a non-transient computer-readable storage medium storing computer program instructions. When the computer program instructions are executed on the processor 420 of the electronic device 400, the processor 420 executes the mobile advertising detection method based on multimodal proxy user interface navigation described in the above embodiments.

[0101] When the computer program instructions are executed by the processor 420, the following operation steps are specifically implemented: controlling the offline profile building subsystem 100 to perform multi-layer static analysis and lightweight dynamic detection on the target mobile application to generate a prior knowledge base containing screen prior data, slot prior data, trigger prior data and network prior data; controlling the UI navigation subsystem 200 guided by multimodal reasoning to load the prior knowledge base and start the online detection loop; and controlling the memory-driven runtime optimization subsystem 300 to retrieve and reuse historical advertising trigger experience across applications.

[0102] During the offline profile construction process, computer program instructions cause processor 420 to call static analysis unit 110 to parse the mobile application's manifest file, layout resource file, and bytecode file, extracting permission features, component registration information, ad container location, and application programming interface call attribution relationships. Simultaneously, computer program instructions cause processor 420 to call dynamic probing unit 120 to perform random traversal tests, construct a coarse-grained UI transition graph, and extract network request features that are temporally correlated with user interactions.

[0103] During online navigation, computer program instructions cause processor 420 to call UI unified perception module 210 to identify the rendering type of the current interface and generate component representation data in a unified format through accessibility service interface or hybrid vision detector. Computer program instructions also cause processor 420 to call context state management module 220 to maintain the UI transition graph and short-term interaction history buffer, and call multimodal reasoning agent module 230 to construct structured prompts containing system instructions, current interface observations, prior and policy context, and past experience. Processor 420 inputs the structured prompts into a large language model and parses the output of the large language model to obtain interaction action instructions and immediate relevance scores.

[0104] During runtime optimization, computer program instructions cause processor 420 to call experience retrieval module 330 to calculate the cosine similarity between the current interface state feature vector and the historical experience feature vector, and retrieve interaction path summaries that meet the similarity threshold requirements. When an advertisement is successfully triggered, computer program instructions cause processor 420 to call experience extraction module 310 to generate a trigger state fingerprint and interaction path summary, and call vector storage module 320 to associate and store the trigger state fingerprint and interaction path summary in the vector database.

[0105] Non-transient computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, flash drives, or any other media capable of storing program code. Non-transient computer-readable storage media may be located internally to electronic device 400 or connected to electronic device 400 as external storage devices.

[0106] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A high-efficiency mobile advertising detection system based on multimodal proxy user interface navigation, characterized in that, include: The offline profiling subsystem is configured to perform static feature analysis and dynamic random probing on the target mobile application, and generate a prior knowledge base containing screen prior data, slot prior data, trigger prior data and network prior data. The multimodal reasoning-guided UI navigation subsystem is configured to generate unified component representation data using the UI unified perception module when the target mobile application is running, and to construct structured prompt words using the multimodal reasoning agent module in conjunction with the prior knowledge base; the multimodal reasoning-guided UI navigation subsystem is also configured to use a large language model to reason about the structured prompt words and output interactive action instructions and advertising relevance scores. The memory-driven runtime optimization subsystem is configured to calculate the similarity between the feature vector of the current interface and the feature vector stored in the historical experience base when the target mobile application is running, retrieve the historical interaction path summary and feed it back to the multimodal reasoning-guided UI navigation subsystem; the memory-driven runtime optimization subsystem is also configured to extract the UI state fingerprint of the current state and the interaction path summary generated by the large language model and store them in the historical experience base when an advertisement is detected.

2. The efficient mobile advertising detection system based on multimodal proxy user interface navigation according to claim 1, characterized in that, The offline profile building subsystem includes a static analysis unit, which is configured to perform the following operations: By parsing the manifest file of the target mobile application, permission declarations and component registration information are extracted to generate the screen prior data; By parsing the layout resource file of the target mobile application and matching the class names of the view nodes with a list of pre-built advertising development kits, the location of the advertising container is identified to generate the slot prior data. By analyzing the bytecode call graph of the target mobile application, the application programming interface calls related to the advertisement are traced back to the specific Activity component, and an attribution mapping relationship is established to generate the trigger prior data.

3. The efficient mobile advertising detection system based on multimodal proxy user interface navigation according to claim 1, characterized in that, The offline profile construction subsystem also includes a dynamic detection unit, which is configured to perform the following operations: The target mobile application is driven to perform random traversal tests to construct a coarse-grained UI transformation graph; runtime system logs are captured, and network request records that match the preset advertising feature library are extracted; Based on the timestamp alignment algorithm, a causal relationship is established between the network request record and the user interaction event within the preset time window to generate the network prior data.

4. The efficient mobile advertising detection system based on multimodal proxy user interface navigation according to claim 1, characterized in that, The UI Unified Awareness module is configured to perform cross-rendering consistency processing: When the current interface is detected to be at the standard view level, the accessibility service interface is called to extract the class name, resource identifier and text content of the control as the unified component representation data; When the current interface is detected as a canvas rendering view, the coordinates of the control area are identified using an object detection model, and the semantic description text of the control area is generated using a visual language model. The coordinates and the semantic description text are then integrated into the unified component representation data.

5. The efficient mobile advertising detection system based on multimodal proxy user interface navigation according to claim 1, characterized in that, The multimodal reasoning-guided UI navigation subsystem also includes a context state management module, which is configured as follows: Maintain a UI transition graph, where nodes represent Activity component states and edges represent user interaction events; Maintain a short-term interaction history buffer to store the most recent preset number of interaction records; calculate the length of the time window in which the user is currently continuously in the same Activity component state; when the length of the time window exceeds a preset stagnation threshold, generate a stagnation state marker and write it into the current exploration state data.

6. The efficient mobile advertising detection system based on multimodal proxy user interface navigation according to claim 1, characterized in that, The multimodal reasoning agent module is configured to construct the structured prompt words, which contain the following four data blocks: The system instruction block defines the role parameters, task objectives, and output format specifications of the large language model. The current interface observation block contains the serialized unified component representation data; The prior and policy context block contains data from the prior knowledge base that matches the current interface, as well as the current local UI transformation graph data; The past experience block contains a summary of the historical interaction paths retrieved by the memory-driven runtime optimization subsystem.

7. The efficient mobile advertising detection system based on multimodal proxy user interface navigation according to claim 1, characterized in that, The multimodal inference agent module is also configured to maintain path runtime estimates: Receive the ad relevance score output by the large language model for the current time step; calculate the running estimate for the current time step by combining the running estimate from the previous time step with the ad relevance score using an exponential moving average algorithm; when the running estimate is lower than a preset abandonment threshold, generate a path backtracking instruction or a restart instruction.

8. The efficient mobile advertising detection system based on multimodal proxy user interface navigation according to claim 1, characterized in that, The memory-driven runtime optimization subsystem includes an experience extraction module, which is configured as follows: Extract the control class name set, resource identifier set, and text content set of the state before the advertisement is triggered, and concatenate them according to the preset spatial sorting rules to generate the UI state fingerprint; Construct prompt words containing causal reasoning instructions, input the operation sequence that leads to ad triggering into the large language model, and generate the interaction path summary.

9. The efficient mobile advertising detection system based on multimodal proxy user interface navigation according to claim 8, characterized in that, The memory-driven runtime optimization subsystem further includes a vector storage module, which is configured as follows: The UI state fingerprint is mapped to a high-dimensional feature vector using a text embedding model; the high-dimensional feature vector is normalized; a vector index is constructed using a hierarchical navigable small-world indexing algorithm or an inverted file indexing algorithm; and the high-dimensional feature vector is associated with and stored with the interaction path summary.

10. The efficient mobile advertising detection system based on multimodal proxy user interface navigation according to claim 8, characterized in that, The memory-driven runtime optimization subsystem further includes an experience retrieval module, which is configured as follows: The UI state fingerprint of the current interface is mapped to the current feature vector using a text embedding model; the cosine similarity between the current feature vector and the feature vector stored in the historical experience database is calculated; when the cosine similarity exceeds a preset matching threshold, the corresponding interaction path summary is extracted and sent to the multimodal reasoning agent module.

Images