Mobile application behavior characterization method based on page semantic graph
By building a page semantic graph and integrating the user interface and code information of mobile applications, the problem of difficult to combine interfaces and code in the existing technology is solved, and more efficient security analysis and privacy support is achieved.
Patent Information
- Application Number
- CN202410948899.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-12-03
AI Technical Summary
Existing mobile application characterization methods cannot provide both high fidelity and high availability, making it difficult to seamlessly combine user-visible user interfaces with underlying code, resulting in difficulty in security and privacy analysis.
Using a method based on page semantic graph, we use a large language model to generate a page semantic summary, and build a page semantic graph, integrate user interface and code information, and provide a new representation method.
Improves the understanding and analytical performance of mobile applications, enables a more comprehensive and in-depth understanding of applications, enhances security analysis capabilities, and provides more accurate security and privacy support.
Smart Images

Figure CN119066651A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of mobile security technology, and in particular relates to a behavior characterization and analysis method for mobile applications. Background Art
[0002] Current research on mobile application security and privacy increasingly relies on program analysis and artificial intelligence models to understand and analyze applications. For example, when detecting malware, it is common to use static or dynamic program analysis to extract features from mobile applications, and then input these features into artificial intelligence models for classification. However, as mobile applications continue to evolve, their user interfaces (UIs) and underlying codes are becoming more and more complex, which poses a huge challenge to programmatically understanding and analyzing the UIs and codes of these applications.
[0003] Currently, different application characterization methods have been proposed, including permission usage patterns, code representation techniques, and user interface representation methods (UI). One challenge encountered by these representation methods is how to seamlessly combine the semantics presented in the upper-level user interface visible to users with the underlying code responsible for executing the core logic. As mobile applications develop towards rich functions and complex user interactions, it becomes crucial to analyze both the user interface and the code. This dual consideration is particularly important in security analysis, because understanding the interaction between what the user sees and what happens behind the scenes can reveal potential security and privacy impacts. It can be seen that existing application characterization methods cannot provide high fidelity and high usability at the same time.
[0004] Recently, the rapid development of large language models (LLMs) has opened up new avenues for improving the representation of applications. These models are capable of handling complex natural language tasks, not limited to text generation and translation, but also show excellent capabilities in understanding and processing programming languages and user interfaces. Previous studies have shown that LLMs can effectively understand the user interface (UI) and code of mobile applications through fine-tuning or hint engineering. This understanding is not limited to the surface structure, but can also go deep into the logic and functional level of the application, providing a unified and comprehensive semantic view of the application. This view combines information from the UI and the underlying code, which helps to analyze and understand the application more comprehensively and deeply, thereby providing important technical support for developers and researchers, enabling them to optimize and innovate mobile applications more effectively. Summary of the invention
[0005] The purpose of the present invention is to provide a method for characterizing mobile application behavior based on a page semantic graph, so as to improve the performance of mobile application understanding and analysis tasks and alleviate the security issues and threats faced by the mobile system ecosystem. Specifically, a page semantic graph is constructed to represent mobile applications for in-depth understanding and analysis.
[0006] The method for representing mobile application behavior based on page semantic graph provided by the present invention first constructs a page semantic graph to represent the mobile application program so as to perform in-depth understanding and analysis.
[0007] The present invention includes: identifying application pages and extracting UI and code information in the pages; summarizing page semantics and generating summaries using LLM; constructing page semantic graphs based on the transfer relationships between pages and page semantic summaries, providing a new representation method for the behavior analysis of mobile applications.
[0008] The core of this invention is to introduce a new method to represent mobile applications, namely, page semantic graph. The overall construction process is as follows: Figure 1 This approach integrates code and user interface elements, which is different from existing methods that only use code or only use user interface. The overall idea includes two main steps: first, convert the mobile application into a graph format to incorporate the semantic details in both the user interface and the code; then, use the semantic graph to perform various analysis tasks on the mobile application with a comprehensive understanding of the application's functions and intentions.
[0009] In the present invention, the application page is identified and the UI and code information in the page is extracted. First, all pages in the mobile application, including subpages, are identified. Then, for each identified page, its user interface information, code information and call graph are collected to comprehensively extract the semantic information of the application's UI and code.
[0010] The page summary generation module uses LLM to summarize the page semantics and generate a summary. Specifically, the present invention adopts a prompt engineering method based on thought chain (COT) to enable LLM to generate a page summary that considers user interface and code semantic information.
[0011] The page semantic graph construction module uses static analysis methods to identify all inter-component communications (ICC) between pages, and extends from the traditional communication between activities and fragments to ICC calls in dialogs and Webviews. This extension ensures comprehensive coverage of the transition relationships of all pages and sub-pages. After collecting the semantics of each page and its conversion relationship, the present invention constructs the final page semantic graph.
[0012] Specifically, the method for representing mobile application behavior based on page semantic graph provided by the present invention includes three steps: extracting page information based on static analysis, generating page semantic information summary based on a large language model, and constructing a page semantic graph based on page transfer relations, wherein:
[0013] (i) Page information extraction based on static analysis, which aims to capture the semantic essence of the UI and underlying code of each page in the mobile application by analyzing the user interface (UI) elements and underlying code of each page in the mobile application;
[0014] (2) Page semantic information summary generation based on the big language model is to use the big language model to semantically summarize the UI elements and codes in the page semantic graph and generate a page semantic information summary to describe the function and intention of each page;
[0015] (iii) Page semantic graph construction based on page transfer relationships, using the transition relationships between pages to complete the construction of the page semantic graph, to help subsequent analysis tasks fully understand the functions and intentions of the application.
[0016] Further:
[0017] (1) The page information extraction includes the following steps:
[0018] (a) Page identification, i.e. identifying all pages and sub-pages in a mobile application for further information extraction;
[0019] The goal of page identification is to identify all pages and sub-pages in an Android application. Pages are activities in a UI hierarchical tree format. Activities provide a visual representation of the user interface and define the behavior of the user interface through various UI elements and underlying code, and are widely used in state-of-the-art UI modeling work. In addition, Android applications also tend to use sub-page components such as fragments, dialog boxes, and WebViews to enhance the complexity and flexibility of the user interface. To identify all pages, the application's code and resources need to be analyzed, and various components related to each page need to be found, including activities, fragments, dialog boxes, and WebViews.
[0020] (b) UI information extraction, i.e. extracting the hierarchical structure of UI elements for each page to reflect the structure and semantics of the page UI;
[0021] After identifying all pages and subpages, the present invention extracts the hierarchical tree of UI elements for each page and subpage to reflect the structure and semantics of the page user interface. For each user interface element in the hierarchy, its key attributes need to be collected, such as resource ID, input type, and user-perceivable text (such as title and prompt). These attributes are crucial for capturing the interactive and visual aspects of the user interface.
[0022] (c) Code information extraction, that is, extracting the underlying code corresponding to the UI elements for each page;
[0023] Specifically, for each page, the goal is to extract the underlying code corresponding to the UI elements. These codes are mainly divided into three categories: (1) Lifecycle functions: Android defines lifecycle functions for activities and fragments, which are crucial in managing the state and behavior of these components. (2) Event callbacks: UI elements that users interact with are associated with event response functions, which trigger actions when users interact with them. (3) Other functions: These include various auxiliary functions provided in the application, which support the functionality of the application.
[0024] The present invention utilizes existing static analysis tools to generate the entry point of each page and extracts the call chain of the target function to extract the underlying code information of the page. In order for the subsequent page semantic analysis to accurately associate UI information with code information, the present invention utilizes the association between the UI elements of user interaction and their attached callback practices to establish a mapping between UI objects and codes.
[0025] In addition, the process of establishing a mapping encounters the difficulty of developers assigning a single event listener to multiple UI elements. The present invention addresses this difficulty by pruning irrelevant paths in callbacks, which is based on execution flow tracking associated with element IDs.
[0026] Further:
[0027] In the page recognition, in addition to identifying the main interface represented by the activity, the page recognition also needs to identify the layout statically embedded in the activity and all dynamically loaded sub-pages, so as to ensure that all pages in the mobile application can be accurately identified and provide comprehensive semantic information for subsequent analysis.
[0028] In the UI information extraction, it is necessary to collect key attributes of UI elements, such as resource ID, input type and user-perceivable text (such as title and prompt). These attributes are crucial for capturing the interactive and visual aspects of the user interface. For the UI information of subpages, it is necessary to pay attention to solving the problem that subpage elements are difficult to obtain. The present invention can accurately find out the specific user interface elements and corresponding codes related to each type of subpage by manually checking the official documents of the subpages.
[0029] In the code information extraction, the corresponding underlying code needs to be extracted for each UI element, and a mapping between the UI object and the code needs to be established to provide correct information for subsequent modules. The process of establishing the mapping needs to solve the problem that developers assign a single event listener to multiple UI elements.
[0030] (II) Page semantic information summary generation includes the following steps:
[0031] (a) Page content fusion, that is, generating a summary for each function and integrating the UI and code;
[0032] Page content fusion is the semantics of aggregated UI elements and their corresponding codes. The present invention first generates a semantic summary for each function, and then interweaves the UI elements with the corresponding code summaries so that the large language model can understand them.
[0033] The present invention extracts information from three types of code: lifecycle functions, event callbacks, and other functions, and generates function summaries for the above three types of code. Considering that a function will call other functions, the present invention generates function summaries from bottom to top along the call graph to solve this difficulty. The present invention adopts a one-shot method to guide a large language model to generate a summary for each function. The core of this method is to show an example of a function summary to LLMs in a prompt, so as to train it to generate summaries for subsequent functions. In addition, in order to improve the efficiency of the code summary generation process, the batch processing method of the present invention reduces the number of single summaries that need to be generated and facilitates the generation of code summaries.
[0034] After obtaining the function summary, the UI and code of the fused page need to be aligned. The present invention mainly aligns them according to the lifecycle functions and user interaction logic of the components. In addition to the lifecycle functions, the fused content further describes the interaction between the UI elements and the user in detail. This is achieved by integrating the summary of the event callbacks associated with each UI element, effectively mapping the user interaction to its corresponding code response. By adopting this structured approach, the present invention enables LLMs to understand the page through its natural workflow and effectively integrate code and UI semantics.
[0035] (b) Page summary generation, i.e., generating a summary for each page based on the fused content;
[0036] The present invention specifically adopts a prompt engineering method called Chain of Thought (COT) (Wei, Jason, et al. "Chain-of-thought prompting elicits reasoning in large language models."). This method aims to simulate the logical process of human thinking through a series of prompts, guiding LLMs to generate required and relevant content. COT is currently widely used and has shown effectiveness in different analytical tasks.
[0037] The present invention uses the COT method to guide LLM to consider the following specific factors when summarizing a page: (1) Consider code semantic information and page title to determine the main function of the page. (2) Consider all UI elements on the page. (3) Determine which functions can interact with the user through different UI elements. (4) Identify all content present on the page to determine whether there are any sensitive behaviors. These questions can help LLM fully understand the page by considering semantic information from the code and UI.
[0038] To help the LLM follow the thought chain process, the present invention includes a practical example in the first step of the prompt. This example shows how to answer the questions posed in the thought chain sequence. For example, when the present invention asks which UI elements are interactive for the user and what functions they trigger, the LLM is guided to answer in the form of specific functions that correspond to the UI elements that the user can interact with. Through this single example, the LLM is more able to follow the specified thought chain and improve the accuracy and relevance of its response.
[0039] (c) Page tag generation, that is, generating keywords for each page to tag the page to concisely reflect the semantic information of the page, including functions and features. These tags can facilitate faster, semantic-based understanding and search of these applications. The present invention first constructs a vocabulary for page tags by combining official documents and previous taxonomies and research; by considering terms from different sources, the generated tags can cover different aspects of the application semantics, providing users with meaningful hints about the content and purpose of each page. After obtaining the vocabulary, the "Term Frequency-Inverse Document Frequency (TF-IDF)" of the words in the previously generated page summary is calculated to obtain page tags. TF-IDF is used to reflect the importance of a word in a document relative to the document corpus.
[0040] Further,
[0041] In the page content fusion, it is necessary to use large language models (LLMs) to generate semantic summaries for each function, and then interweave UI elements with corresponding code summaries so that the LLM understanding of the subsequent page summary generation part is achieved. And considering that a function may call other functions, the present invention generates function summaries from bottom to top along the call graph to solve this difficulty.
[0042] In the page summary generation, the present invention uses the COT method to guide LLM to consider specific perspectives when summarizing the page, which can help LLM fully understand the page by considering semantic information from code and UI. Moreover, by using one-shot examples, LLM is more able to follow the specified chain of thinking, improving the accuracy and relevance of its response.
[0043] (iii) Construction of the page semantic graph, specifically using static analysis technology to identify all inter-component communications (ICC) between pages, and use this as the transfer relationship between pages. Based on previous studies that mainly focused on ICC between activities and fragments, the present invention expands the scope of analysis to include ICC calls in dialog boxes and Webviews. This extension ensures that this step can fully cover the transition relationships of all pages and sub-pages. This step constructs the final page semantic graph (PSG) based on the collected semantic information of each page and its transition relationship. In this graph, each node represents a page and its semantics, and each edge represents a transition between two pages. The construction of the PSG provides a high-level overview of the application structure and functionality, thereby enhancing the ability to analyze and understand the underlying functions and intent of the application.
[0044] The technical features and beneficial effects of the present invention mainly include:
[0045] (1) This paper introduces a new method for representing mobile applications, called Page Semantic Graph (PSG), which takes into account the semantic information obtained from the UI and code of the mobile application and leverages the capabilities of program analysis and large language models.
[0046] (2) We design a prototype system that can automatically build PSGs for mobile applications by seamlessly integrating UI and code information and adopting the Chain of Thought (COT) approach to guide LLMs to generate semantically rich summaries for each page.
[0047] (3) The application characterization method proposed in this invention can improve the performance in tasks related to mobile application understanding and security analysis. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 Flowchart for the PSG construction of the present invention.
[0049] Figure 2 Generates a workflow diagram for the module for page summaries. DETAILED DESCRIPTION
[0050] The present invention is further described below by way of embodiments in conjunction with the accompanying drawings.
[0051] (I) Page Information Extraction
[0052] The present invention proposes an analysis tool that can build PSG according to the APK file of a mobile application. First, the present invention develops a static analysis tool for extracting UI interface and underlying code based on Soot and FlowDroid, and implements pointer analysis of UI-related objects through Boomerang. Using this static analysis tool, the manifest file of the application is parsed to identify all activities corresponding to the page, and the XML file generated by the activity in the page initialization phase is analyzed to identify the sub-pages statically defined in the layout. The present invention also uses a lightweight pointer analysis method to track the sub-page objects dynamically instantiated in the activity code, thereby identifying all dynamically loaded sub-pages. For each page, a hierarchical tree of UI elements is extracted to reflect the structure and semantics of the page UI. After the extraction is completed, the user interface hierarchy tree will be converted into HTML format for more direct analysis in subsequent stages. The page information extraction module uses existing static analysis tools to generate the entry point of each page, which usually exists in the form of a "virtual main method". The method includes calling the above-mentioned different types of code. This module then extracts the subsequent call chains of these entry points from the application's call graph and maps the extracted code information to the corresponding UI elements by parsing the layout files and callbacks registered in the code.
[0053] (II) Page semantic information summary generation
[0054] This step first generates a function summary. Since a function will call other functions, the present invention generates a function summary from bottom to top along the call graph. The bottom-up method includes the following steps. First, for the function that needs to generate a summary, mark it as a root node and obtain its call graph; then remove the loop in the call graph in a depth-first manner; then generate function summaries for all leaf nodes and nodes on the call graph to the root node. In addition, when creating summaries for non-leaf nodes in the call graph, the present invention includes the summaries of its child nodes as input. In order to generate a summary for each function, the present invention adopts a one-shot learning method to guide large language models (LLMs) to complete this task.
[0055] Next, the UI and code of the page are fused, aligning them according to the component lifecycle functions and user interaction logic. The fused content starts with the page name, which is extracted from the code that defines "Activity". Next is the function summary of the onCreate method, as it represents the entry point of the page. Then, the UI elements and their attributes in HTML format are extracted, where each UI element is assigned a unique identifier to maintain its uniqueness in the page structure. Next, the fused content continues to track the execution flow of the page lifecycle, including methods such as onResume() and onPause(). The fused page content ends with a summary of the onDestroy() method, marking the end of the page lifecycle.
[0056] Next, a summary is generated for each page based on the fused page content. The Chain of Thought (COT, Wei, Jason, et al. "Chain-of-thought prompting elicits reasoning in large language models." Advances in neural information processing systems 35(2022): 24824-24837.) method is used to guide LLMs to generate a semantic summary of the page, which mainly includes three steps: 1. Provide a prompt to LLMs, including a task description, a one-time example, and the fused content of the page. 2. Ask a series of questions designed to mimic user thinking and help LLMs gradually clarify the structure and semantics of the page 3. Ask a summary question (Final Q) to prompt LLMs to summarize the function and intention of the page. LLM's answer to this question forms a comprehensive page summary.
[0057] For the generation of page labels, we first constructed a vocabulary for page labels by combining official documents and previous taxonomies and studies. Specifically, we first collected the user interface (UI) and user experience (UX) categories from previous studies on semantic classification of mobile application design, and then obtained the "capability" category from the built-in intent model of the Android official documentation, which represents different functional operations of Android applications. Then, we collected the "security and privacy" category by referring to previous security analysis work, which includes concepts related to permissions, data sources, and data sinks. Finally, we obtained a vocabulary containing 293 terms. Based on this vocabulary, we tokenized the text of each page summary into individual words, and then restored these words to their basic forms and filtered out words that were not in the predefined page label vocabulary. Then, we calculated the TF-IDF score of each preprocessed word, and used the top three words as the label of the page.
[0058] (III) Page semantic graph construction
[0059] Based on the static analysis tools mentioned above, the transition relations of all pages and sub-pages can be extracted. By collecting the semantics of each page and its transition relations, the final page semantic graph (PSG) can be constructed. In this graph, each node represents a page with its semantics, and each edge represents the transition between two pages.
[0060] (IV) Application and verification of the method of the present invention
[0061] The static analysis tool runs on an Ubuntu server equipped with a 30-core CPU (2.80GHz) and 241GB of memory. For the page summary generation module, the present invention uses ChatGLM3-6B, an open source pre-trained LLM for generating summaries of function code and fused page content. The present invention selects a version containing 6 billion parameters and runs it on a server equipped with 2 GeForce RTX 3090 GPUs. Three experiments show that the characterization method implemented by the present invention is effective in various application understanding and security analysis tasks, including application description generation, special malware detection, and authentication security analysis.
[0062] The description of an application is the main reference for users when browsing the application market. A well-crafted application description is required to effectively help users understand the features and functions of the application. In addition, an accurate and complete description of sensitive behaviors helps users choose whether to use the application based on their privacy concerns. Therefore, description generation is a classic research topic in both software engineering and security. To this end, the characterization method proposed in this paper is used to help generate effective mobile application descriptions so that users can fully understand the behavior of the application. The present invention selects the most advanced description generation method, namely DescribeCtx (Yang, Shao, et al. "Describectx: context-aware description synthesisfor sensitive behaviors in mobile apps." Proceedings of the 44th International Conference on Software Engineering. 2022.) as a benchmark, which uses a seq2seq model to generate descriptions for sensitive behaviors in applications. DescribeCtx uses two metrics, namely BLEU and ROUGE-L, to measure the quality of the generated descriptions, which are also used in this experiment. Specifically, the BLEU score calculates the percentage of grams in the generated description that appear in the reference sentence, while the ROUGE-L metric considers the longest common subsequence between the reference sentence and the generated description. In this case, the present invention generates a page semantic graph for the target application and extracts all page summaries related to sensitive behaviors of the same permission group used in DescribeCtx. The sentences describing sensitive behaviors in the page summaries are also filtered using the same keywords in DescribeCtx, and these sentences are trained on the seq2seq model to generate the final description of the target application. The final results show that the page semantic graph-based method achieves better performance than the baseline method in all permission groups, which is reflected in higher BLEU and ROUGE-L scores (at least 52% higher).
[0063] Malware detection is another classic task in the security field, which aims to detect malware from real-world applications that contain both benign and malicious malware. Recently, DeUEDroid proposed to detect a special type of malware, namely underground economy malware (UEWare for short). This type of malware is characterized by deceiving users on the application user interface, leading users to make quick decisions without thinking, which may lead to fraud. There is often no obvious malicious payload in the code of UEWare, so the detection of this type of malware should consider both the user interface and the behavior of deceiving users in the code. We evaluate how to use page semantic graphs to help detect this special type of malware. Specifically, DeUEDroid (Chen, Zhuo, et al. "Deuedroid: Detecting underground economy apps based on utg similarity.") is selected as a baseline, which can statically build the user transition graph (UTG) of the application and apply a graph embedding method to represent the UTG, and then use a self-supervised learning method to detect and classify UEware based on the similarity of UTG features. DeUEDroid provides a UEWare dataset containing 1,700 applications collected from the real world. These applications include 310 gambling applications, 441 pornographic applications, 97 financial applications, and 852 general applications. We then use the idea of building a page semantic graph for each application to capture the user interface and code behavior to detect and classify UEWare from this dataset. Specifically, we propose a GCN-based detection and classification method. First, a page semantic graph is built for each application. For each page summary in the application, the GloVe model (Pennington, Jeffrey, Richard Socher, and Christopher D. Manning. "Glove: Global vectors for word representation." Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). 2014.) is used for word embedding, and then integrated with an LSTM containing a self-attention mechanism to generate a feature vector for each page. After that, the PSG with the processed vector is put into a graph convolutional network (GCN) for detection and classification. In this process, the same dataset partitioning scheme as DeUEDroid is adopted, that is, 70% training set, 20% validation set, and 10% test set.The proposed method achieves an F1 score of 0.9933 in UEWare detection and an accuracy of 99.23% in malware multi-class classification. Considering that the proposed method is not explicitly optimized for UEWare detection, this performance shows that the Page Semantic Graph (PSG) is inherently capable of detecting applications with hidden UI intentions. This effectiveness is attributed to the robust fusion of application UI and code information, and the comprehensive integration of its semantics by PSG.
[0064] Current applications tend to use increasingly secure authentication methods, one common practice being the use of a two-factor authentication (2FA) mechanism, where users are required to provide two factors, such as a password and a text message verification code, to complete authentication. However, in the event that a user forgets their password or changes their phone number, the application typically provides a "recovery process" that allows the user to recall the forgotten password and change the registered email or phone number. Previous work (cite{hammann2019user}) has studied backdoors in this authentication process, meaning that it is easier to access an application's account using the recovery process than using the main 2FA process. Specifically, when an attacker controls one authentication factor of a victim's account, if he / she is able to exploit this one factor and also know or change another factor through the recovery process, then the attacker can eventually pass 2FA as the victim. The idea of using PSG to quickly detect authentication backdoors consists of two main steps: 1) PSG construction and authentication analysis. 2) Authentication factor comparison. First, we build a PSG for each application and use the page semantics and transition relationships described in the PSG to identify the main 2FA process and the recovery process. For the primary 2FA process, login-related pages are located by matching the keyword "login" in the page tags, and all relevant pages involved in the primary authentication are tracked through PSG. The page summary is analyzed to determine whether 2FA is enforced and to identify the two factors involved in the verification process. For the recovery process, we search for pages by matching keywords such as "reset", "forgot", and "change phone number / email". This provides an understanding of the available methods for recovering account access and the factors that can be reset or retrieved. The verification factors identified in the main process and the recovery process are then compared. This comparison aims to determine whether the recovery process allows a single factor to override or control the two factors used in the main 2FA process, thereby identifying potential backdoors.
[0065] Among the 978 2FA applications, we successfully built PSGs for 832 of them (the remaining applications could not be processed due to the limitations of static analysis or the lack of detection of authentication-related information). Among the 832 applications analyzed, PSGs were successfully created for 832 applications. Among the 832 applications analyzed, the method of the present invention found that 112 applications were at risk of 2FA, while the remaining 720 applications were not at risk of 2FA. 10% (i.e., 72 applications) were randomly selected from these 720 applications for manual analysis. Among these 72 applications, 61 (84.7%) had authentication processes extracted from PSG that were consistent with their actual implementations, while the remaining false negatives were mainly attributed to the failure of PSG to extract all user interface information when dynamic pages were loaded. Subsequently, the 112 applications were analyzed and verified in detail through manual inspection. As a result, the present invention successfully confirmed 76 backdoor risks in 72 applications, and the remaining 40 applications were found to have no backdoor risks, which was usually due to their complex recovery processes including out-of-band protection, such as additional manual verification during email-based password recovery, which is beyond the scope of our current method. Despite the limitations, the results prove that the present method has significant efficacy in facilitating the detection of authentication backdoors in real-world mobile applications.
[0066] The experimental results show that the present invention can effectively extract and describe the semantic information of mobile application interfaces, and provide a new representation method for the behavior analysis of mobile applications.
Claims
1. A method for characterizing mobile application behavior based on page semantic graph, characterized in that: First, we build a page semantic graph to represent mobile applications for in-depth understanding and analysis. This includes: identifying application pages and extracting UI and code information from the pages; summarizing page semantics and generating summaries using LLM; building a page semantic graph based on the transition relationship between pages and page semantic summaries, providing a new representation method for the behavior analysis of mobile applications. This is divided into three steps: page information extraction based on static analysis, page semantic information summary generation based on a large language model, and page semantic graph construction based on page transition relationships, where: (i) Page information extraction based on static analysis, which aims to capture the semantic essence of the UI and underlying code of each page in the mobile application by analyzing the user interface (UI) elements and underlying code of each page in the mobile application; (2) Page semantic information summary generation based on the big language model is to use the big language model to semantically summarize the UI elements and codes in the page semantic graph and generate a page semantic information summary to describe the function and intention of each page; (iii) Page semantic graph construction based on page transfer relationships, using the transition relationships between pages to complete the construction of the page semantic graph, to help subsequent analysis tasks fully understand the functions and intentions of the application.
2. The method for characterizing mobile application behavior based on page semantic graph according to claim 1, characterized in that: (1) The page information extraction includes the following steps: (a) Page identification, i.e. identifying all pages and sub-pages in a mobile application for further information extraction; The goal of page recognition is to identify all pages and subpages in an Android application. A page is an activity in a UI hierarchical tree format. An activity provides a visual representation of the user interface and defines the behavior of the user interface through various user interface elements and underlying code. Specifically, by analyzing the application's code and resources, we can find various components related to each page, including activities, fragments, dialog boxes, and WebViews, and achieve recognition of all pages. (b) UI information extraction, i.e. extracting the hierarchical structure of UI elements for each page to reflect the structure and semantics of the page UI; Specifically, after identifying all pages and sub-pages, a hierarchical tree of UI elements is extracted for each page and sub-page to reflect the structure and semantics of the page user interface; for each user interface element in the hierarchy, its key attributes are collected, including resource ID, input type, and user-perceivable text; (c) Code information extraction, that is, extracting the underlying code corresponding to the UI elements for each page; Specifically, for each page, the underlying code corresponding to the UI elements is extracted; these codes are divided into three categories: (1) Lifecycle functions: Android defines lifecycle functions for activities and fragments; (2) Event callbacks: UI elements that users interact with are associated with event response functions, which trigger actions when users interact with them; (3) Other functions: These include various auxiliary functions provided in the application, which support the functions of the application; (II) Page semantic information summary generation includes the following steps: (a) Page content fusion, i.e. generating a summary for each function and fusing the semantics of aggregated UI elements and their corresponding codes; We first generate a semantic summary for each function, and then interweave UI elements with the corresponding code summaries for large language models to understand; specifically: Extract information from three types of code: lifecycle functions, event callbacks, and other functions, and generate function summaries for the above three types of code; use a one-shot method to guide a large language model to generate summaries for each function; After obtaining the function summary, the UI and code of the page are integrated: mainly aligning them according to the component lifecycle functions and user interaction logic; in addition, the integration content includes a detailed description of the interaction between UI elements and users; (b) Page summary generation, i.e., generating a summary for each page based on the fused content; Specifically, the COT hint engineering method is adopted to guide LLM to consider the following specific factors when summarizing the page: (1) consider the code semantic information and page title to determine the main function of the page; (2) consider all UI elements on the page; (3) determine which functions can interact with users through different UI elements; (4) identify all the content present on the page to determine whether there is any sensitive behavior; (c) Page tag generation, i.e., generating keywords for each page to tag the page to concisely reflect the semantic information of the page, including functions and features; Specifically, we first build a vocabulary for page tags by combining official documents and previous taxonomies and research. By considering terms from different sources, the generated tags can cover different aspects of application semantics and provide users with meaningful hints about the content and purpose of each page. After obtaining the vocabulary, we calculate the "term frequency-inverse document frequency (TF-IDF)" of the words in the previously generated page summary to obtain page tags. (iii) Construction of the page semantic graph, specifically using static analysis techniques to identify all inter-component communications (ICCs) between pages and use them as transfer relationships between pages; Specifically, the final page semantic graph (PSG) is constructed based on the collected semantic information of each page and its transition relationship; in this graph, each node represents a page and its semantics, and each edge represents the transition between two pages.
3. The method for characterizing mobile application behavior based on page semantic graph according to claim 2, characterized in that: In the page information extraction step: In the page identification, in addition to identifying the main interface represented by the activity, the page identification also includes identifying the layout statically embedded in the activity and all dynamically loaded sub-pages, so as to ensure that all pages in the mobile application can be accurately identified and provide comprehensive semantic information for subsequent analysis; In the UI information extraction, key attributes of UI elements are collected, including resource ID, input type and user-perceivable text; for the UI information of sub-pages, the official documents of sub-pages are manually checked to accurately find specific user interface elements and corresponding codes related to each type of sub-page; In the code information extraction, the corresponding underlying code is extracted for each UI element, and a mapping between the UI object and the code is established to provide correct information for subsequent modules.
4. The method for characterizing mobile application behavior based on page semantic graph according to claim 2, characterized in that: In page semantic information summary generation: In the page content fusion, a large language model is used to generate a semantic summary for each function, and then the UI elements are interwoven with the corresponding code summaries to facilitate LLM understanding of the subsequent page summary generation part. In the page summary generation, the COT method is used to guide the LLM to consider specific factors when summarizing the page, which helps the LLM to fully understand the page; by considering semantic information from the code and UI, and using one-shot examples, the LLM is more able to follow the specified chain of thinking, thereby improving the accuracy and relevance of its response.
Citation Information
Cited By
Report generation apparatus and method
WO2026135163A1
Issue reporting method, computer program therefor, and computer-readable storage medium storing computer program
WO2026135284A1
