An automated evaluation method, system, and device for Android permission control mechanism

Abstract

The present invention discloses an automated evaluation method for an Android permission control mechanism, comprising the following steps: S1: initializing a test environment and establishing communication with a device to be tested; S2: scanning the operating system permission list of the device to be tested, generating a test task tree, and initiating a test process; S3: pre-configuring the target permission to be tested; S4: triggering the target permission behavior to be tested; S5: reading the current permission status of the operating system and performing post-operation status verification; S6: analyzing the test results and generating a test conclusion; S7: updating the execution status of the test task tree, selecting the next test task to continue executing, and outputting a test report until all test tasks are completed. The test process is automated, fully covering boundary scenarios to ensure test completeness; saving time and being less prone to errors; having high test efficiency, meeting the test requirements of rapid iteration of batch devices, enabling structured storage and intelligent comparison with expected rules, and high root cause analysis efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of Android system access control, and in particular relates to an automated evaluation method, system and device for an Android permission control mechanism. Background Art

[0002] When evaluating the security capabilities of Android operating system control mechanisms, the "permission tool + manual testing" model is currently widely used. This model triggers permission request cases by manually calling the permission interface testing tool and simulates users managing application permissions in the terminal device's system settings, including granting, revoking, or restricting permissions. This model then manually evaluates whether the permission control mechanism for the application meets expectations and whether there are any defects. This model has the following four major shortcomings:

[0003] 1. Strong reliance on manual experience and low test standardization: Android operating system firmware customizations are numerous and fragmented, with discrepancies between user interfaces (UIs) and operating system versions, and the system is frequently updated and changing. Existing technologies rely on testers to manually configure permission states, execute test cases, and manually judge the results. This requires testers to have a deep understanding of the permission management rules and UI features of each system version. However, manual operation can easily lead to problems such as incorrect permission configuration, missed test case execution, and subjective bias in result judgment due to differences in experience. This makes the testing process difficult to standardize, and test results are difficult to reproducible.

[0004] Second, the testing process lacks dynamic adaptability and coverage of limited scenarios: The existing "permission tool + manual testing" model requires pre-written static test cases, making it impossible to dynamically adjust testing strategies based on the characteristics of the system version being tested. For example, different Android versions have different runtime request mechanisms for dangerous permissions (such as dynamic pop-up and floating window permissions). Manual testing cannot quickly adapt to the changes in permission management logic across multiple versions, resulting in insufficient coverage of edge cases (such as multiple permission grants / revocations and combined permission conflicts), making it difficult to ensure test completeness.

[0005] 3. Lack of automated intelligent evaluation and inefficient manual testing: Testing permission control mechanisms across multiple models and operating systems requires repetitive execution of numerous operations, including permission configuration, application installation, and use case triggering. Existing technologies rely on manual, step-by-step operations, which are time-consuming and error-prone. This is particularly true in continuous testing scenarios (such as permission state transition stress testing), where high-frequency, long-term automated execution is impossible. This results in low testing efficiency and makes it difficult to meet the rapid iteration requirements of batch device testing.

[0006] Fourth, insufficient test result recording and analysis capabilities: Manual testing relies on subjective recording of permission control behaviors (such as pop-up content and permission status synchronization delays), lacking automated monitoring for abnormal events (such as unauthorized access and silent authorization). Furthermore, test logs are mostly discrete text records, making them difficult to store in a structured manner and intelligently compare against expected rules. This reduces the efficiency of defect location and root cause analysis. Summary of the Invention

[0007] Based on this, it is necessary to propose an automated evaluation method for the Android permission control mechanism to address the problems of low test standardization, limited coverage scenarios, low efficiency of manual testing, and insufficient ability to record and analyze test results.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] An automated evaluation method for an Android permission control mechanism, comprising:

[0010] S1: Initialize the test environment and establish communication with the device under test;

[0011] S2: Scan the operating system permission list of the device under test, obtain the permission information of the device under test, build a permission detection list, build a multi-dimensional test task, preset a test strategy, generate a test task tree, and start the test process;

[0012] S3: Read the current permission status of the operating system, set the target permission to be tested to a specified status, and pre-configure the target permission to be tested;

[0013] S4: Triggering the target permission behavior to be tested, monitoring the UI changes of the operating system in real time, detecting permission request pop-ups, and automatically selecting the corresponding option according to the preset test strategy when a permission request prompt is detected. If a non-standard permission prompt is detected, using image recognition and natural language processing technology to understand the prompt content and make a selection;

[0014] S5: Read the current permission status of the operating system, perform post-operation status verification, compare it with the expected permission status, verify the availability status of the operating system function and record changes in the permission status;

[0015] S6: Analyze test results, identify abnormal conditions during the test, and generate test conclusions;

[0016] S7: Update the execution status of the test task tree, select the next test task to continue execution, until all test tasks are completed, and output a test report.

[0017] In some embodiments, obtaining permission information of the device under test in S2 further includes collecting system-level and application-level permissions.

[0018] In some embodiments, constructing the multi-dimensional test task in S2 includes generating a permission type dimension, an authorization status dimension, an operation scenario dimension, and a permission dependency dimension.

[0019] In some embodiments, reading the current permission status of the operating system in S3 further includes verifying whether the pre-configuration of the target permission to be tested is successful. If the verification fails, an error report is sent to retry or adjust the pre-configuration strategy.

[0020] In some embodiments, the S4 further includes recording the operation execution process and application response of the operating system of the device under test.

[0021] In some embodiments, the test conclusion in S6 includes: pass or fail, and an exception description.

[0022] In some embodiments, the S7 also includes automatically adjusting the test strategy if an anomaly or potential risk is detected, increasing the depth and breadth of the test, summarizing all test conclusions and automatically generating an optimization plan, and proposing repair suggestions.

[0023] An automated evaluation system for Android permission control mechanisms, including:

[0024] Startup module: used to initialize the test environment and establish communication with the device under test;

[0025] Task planning agent: used to scan the operating system permission list of the device under test, obtain the permission information of the device under test, build a permission detection list, build multi-dimensional test tasks, preset test strategies, generate a test task tree, and start the test process;

[0026] The permission control agent is used to read the current permission status of the operating system, set the target permission to be tested to a specified status, and pre-configure the target permission to be tested;

[0027] Use case execution agent: used to trigger the target permission behavior to be tested, monitor the UI changes of the operating system in real time, detect permission request pop-ups, and automatically select the corresponding option according to the preset test strategy when a permission request prompt is detected. If a non-standard permission prompt is detected, image recognition and natural language processing technology are used to understand the prompt content and make a selection;

[0028] Request-response agent: used to read the current permission status of the operating system, perform post-operation status verification, compare it with the expected permission status, verify the availability status of the operating system function and record changes in the permission status;

[0029] Result Analysis Agent: used to analyze test results, identify abnormal situations during the test process, and generate test conclusions;

[0030] Evaluation module: used to update the execution status of the test task tree, select the next test task to continue execution until all test tasks are completed, and output a test report.

[0031] An automated evaluation device for an Android permission control mechanism includes a processor and a memory, wherein the memory stores a computer program that can be executed by the processor to implement the method described in any one of the above embodiments.

[0032] Compared with the currently commonly used "authorization tool + manual testing" model, the present invention has the following advantages through automated evaluation: 1. The test process is automated, and the test results are highly reproducible. 2. It can quickly adapt to changes in the logic of multi-version permission management, fully covering boundary scenarios (such as multiple granting / revocation of permissions, combined permission conflicts), and ensuring test completeness. 3. It saves time and is not prone to errors. In continuous testing scenarios (such as permission status switching stress testing), it can achieve high-frequency, long-cycle automated execution, with high test efficiency, and can meet the testing needs of rapid iteration of batch equipment. 4. It can be structured and stored and intelligently compared with expected rules, and the efficiency of defect location and root cause analysis is high. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] in:

[0034] Figure 1 This is a flow chart of an automated evaluation method for an Android permission control mechanism according to the present invention;

[0035] Figure 2 This is a schematic diagram of the architecture of a multi-agent collaborative system driven by a large language model, based on an automated evaluation system for an Android permission control mechanism described in the present invention.

[0036] Figure 3 This is a schematic diagram of an open source large language model of an automated evaluation system for an Android permission control mechanism described in the present invention interacting with various intelligent agents and driving the test process. DETAILED DESCRIPTION

[0037] In order to facilitate the understanding of the present invention, the present invention will be described in more detail below with reference to the accompanying drawings and specific embodiments. It should be noted that when an element is described as "connected" to another element, it can be directly on the other element, or there can be one or more centered elements therebetween. The terms "upper", "lower", "left", "right", "upper end", "lower end", "top" and "bottom" used in this specification indicate an orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second" and the like are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0038] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0039] In order to solve the problems in the existing technology of low test standardization, lack of dynamic adaptation capability of test process, limited coverage of scenarios, lack of automated intelligent evaluation, low efficiency of manual testing, and insufficient ability to record and analyze test results, the following will be combined with the attached Figures 1 to 3 An embodiment of the present invention provides an automated evaluation method, system, and device for an Android permission control mechanism.

[0040] The technical solution to achieve the first purpose of the present invention is an automated evaluation method for the Android permission control mechanism. Figure 1 and Figure 2 , Figure 1 This is a flow chart of an automated evaluation method for an Android permission control mechanism according to the present invention; Figure 2 This is a schematic diagram of the architecture of a multi-agent collaborative system driven by a large language model, based on an automated evaluation system for an Android permission control mechanism described in the present invention.

[0041] An automated evaluation method for an Android permission control mechanism, comprising:

[0042] S1: Initialize the test environment and establish communication with the device under test;

[0043] S2: Scan the operating system permission list of the device under test, obtain the permission information of the device under test, build a permission detection list, build a multi-dimensional test task, preset a test strategy, generate a test task tree, and start the test process;

[0044] S3: Read the current permission status of the operating system, set the target permission to be tested to a specified status, and pre-configure the target permission to be tested;

[0045] S4: Triggering the target permission behavior to be tested, monitoring the UI changes of the operating system in real time, detecting permission request pop-ups, and automatically selecting the corresponding option according to the preset test strategy when a permission request prompt is detected; if a non-standard permission prompt is detected, using image recognition and natural language processing technology to understand the prompt content and make a selection;

[0046] S5: Read the current permission status of the operating system, perform post-operation status verification, compare it with the expected permission status, verify the availability status of the operating system function and record changes in the permission status;

[0047] S6: Analyze test results, identify abnormal conditions during the test, and generate test conclusions;

[0048] S7: Update the execution status of the test task tree, select the next test task to continue execution, until all test tasks are completed, and output a test report.

[0049] Specifically, S1: Establish a stable connection with the Android device under test through ADB (Android Debug Bridge) to ensure normal communication and correct device recognition;

[0050] S2: The task planning agent comprehensively scans the operating system version and permission management mechanism characteristics of the device under test, records the system compatibility and special permission management rules in detail, and accurately collects a list of permission types supported by the device, covering common permissions, dangerous permissions, special permissions, etc., and marks the frequency of use and sensitivity of permissions. It builds a more complete structured permission detection list. In addition to recording the default status, grouping, and related APIs of each permission, it also supplements the dependency and potential impact of the permissions. Based on the permission detection list, it generates a multi-dimensional test task tree to ensure comprehensive test coverage.

[0051] The task planning agent selects a specific test task according to the test task tree, officially starts the test process, and sends the permission pre-configuration instructions to the permission control agent;

[0052] S3: The permission control agent sets the target permission to a specified state through the API or simulated UI operation of the operating system of the device under test. After the permission control agent completes the configuration, it verifies itself and reads the permission state of the operating system to verify whether the pre-configuration is successful;

[0053] When the verification is successful, the authority control agent reports to the task planning agent that the preset environment is ready;

[0054] S4: ① The task planning agent sends test instructions to the use case execution agent. After receiving the test instructions, the use case execution agent calls the corresponding API or performs specific operations to trigger the relevant behavior of the permission;

[0055] ②At the same time, the task planning agent activates the monitoring mode of the request response agent. After the request response agent is activated, it monitors the system UI changes in real time and detects permission request pop-ups. When a permission request is detected, the request response agent automatically selects the corresponding option according to the preset strategy. The options include allow, deny, and allow only this time. If a non-standard or special permission prompt is detected, the request response agent uses image recognition and natural language processing technology to deeply understand the prompt content and make accurate choices, including status inspection and data collection, key filtering and recording of key information.

[0056] S5: After the use case execution agent completes the permission-related actions, the task planning agent notifies the permission control agent to verify the status after the operation. The permission control agent reads the current permission status of the operating system of the device under test and compares it with the expected status;

[0057] S6: Analyze the test results and generate test conclusions. The test includes pass / fail judgment, exception description, severity assessment, etc.

[0058] S7: After completing the test, update the execution status of the test task tree in a timely manner, adjust the test strategy, and select the next test task until all test tasks are completed or the termination condition is met.

[0059] Based on the test results, an optimization plan for the permission control mechanism is automatically generated, specific and feasible repair suggestions are proposed for the problems found, and the test report and improvement suggestions are output to complete the entire testing process.

[0060] Compared with the currently commonly used "authorization tool + manual testing" model, the present invention has the following advantages through automated evaluation: 1. The testing process is automated, and the test results are highly reproducible. 2 It can quickly adapt to changes in the logic of multi-version permission management, fully covering boundary scenarios (such as multiple granting / revocation of permissions, combined permission conflicts), and ensuring test completeness. 3 It saves time and is not prone to errors. In continuous testing scenarios (such as permission status switching stress testing), it can achieve high-frequency, long-cycle automated execution, with high testing efficiency, and can meet the testing needs of rapid iteration of batch equipment. 4 It can be structured and stored and intelligently compared with expected rules, and the efficiency of defect location and root cause analysis is high.

[0061] In one embodiment, obtaining the permission information of the device under test in S2 further includes collecting system-level and application-level permissions.

[0062] Specifically, establish a stable connection with the Android device under test through ADB (Android Debug Bridge) to ensure normal communication and correct device recognition.

[0063] The mission planning agent performs the following core operations in sequence:

[0064] Use the adb shell pm list permissions command to obtain a complete list of permissions supported by the device under test, covering all system-level and application-level permissions.

[0065] Use the adb shell dumpsys package command to deeply analyze and record the initial status of permissions, including default grant, default deny, and related permission attributes.

[0066] Build a sophisticated permission tree structure, clearly display the hierarchical relationship of permissions, clarify the permission group information to which each permission belongs, and provide an intuitive basis for subsequent task planning.

[0067] Leveraging the reasoning capabilities of large language models, based on the constructed permission tree, comprehensive and detailed test tasks are intelligently generated from multiple dimensions.

[0068] The use case execution agent performs the following actions:

[0069] a. Use the start_activity function to activate the application to be tested, ensuring that the application to be tested is in a testable state and that the operating environment meets the test requirements.

[0070] b. Deeply analyze the structured task list generated by the task planning agent to clarify the specific operation steps and goals of each test task.

[0071] c. Use the adb shell pm grant / revoke command to dynamically and accurately configure permission status based on test task requirements, simulating actual scenarios of granting and revoking different permissions.

[0072] d. Use adb shell am broadcast to send a pre-set broadcast notification to trigger the test app's corresponding permission use case call. Ensure that the permission use case is correctly executed in the appropriate scenario. Also, record the key parameters and timestamps during the call.

[0073] The testing process is automated, and test results are highly reproducible. It can quickly adapt to changes in permission management logic across multiple versions, fully covering edge cases (such as multiple permission grants / revocations and combined permission conflicts) to ensure test completeness.

[0074] In one embodiment, constructing the multi-dimensional test task in S2 includes generating a permission type dimension, an authorization status dimension, an operation scenario dimension, and a permission dependency dimension.

[0075] Specifically, the test task tree adopts a multi-dimensional design, and the construction algorithm is as follows:

[0076] Android permission test with the root node set to the device under test

[0077] The first level branches are divided by permission type (normal permission, dangerous permission, special permission)

[0078] The second level branches are divided by permission groups (location, storage, camera, etc.)

[0079] The third level branches are divided by specific permissions (precise location, approximate location, etc.)

[0080] The fourth-level branches are divided by authorization status (not requested, authorized, rejected, this time only, etc.)

[0081] The fifth layer branches are divided by test scenarios (front-end use, back-end use, frequent requests, etc.)

[0082] Depth dimension: For each individual permission, basic test tasks such as granting and revoking are designed separately, and in-depth multi-level permission combination testing is carried out, such as nested combination testing of permissions of different risk levels; at the same time, boundary scenario testing is set up to simulate multiple authorization and revocation operations to test the system's ability to handle complex permission changes.

[0083] Breadth dimension: Comprehensive coverage of multiple permission types, testing for dangerous permissions (such as location, camera, microphone and other key permissions involving user privacy), common permissions (such as network access, vibration and other regular permissions), and special permissions (such as installation of applications, system settings and other permissions with special functions); and covering permission usage tests in different foreground and background scenarios to ensure the effectiveness of the permission mechanism in different operating environments.

[0084] Permission group dimension: Carefully design collaborative testing tasks within the permission group to verify the stability of the collaborative work of various permissions within the same permission group; carry out cross-group permission interaction testing to explore the interactive impact between different permission groups; at the same time, set up permission conflict testing to simulate possible permission conflict scenarios and detect the system's conflict handling strategy.

[0085] The testing process is automated, the test results are highly reproducible, and it can quickly adapt to changes in the permission management logic of multiple versions, fully covering boundary scenarios (such as multiple granting / revocation of permissions, combined permission conflicts), and ensuring test completeness.

[0086] In one embodiment, reading the operating system permission status in S3 also includes verifying whether the target permission pre-configuration is successful. If the verification fails, an error report is sent to retry or adjust the pre-configuration strategy.

[0087] Specifically, the task planning agent carefully selects the preset permission test tasks according to the test task tree and officially starts the test process.

[0088] The task planning agent sends the permission pre-configuration instructions to the permission control agent in a timely manner.

[0089] The permission control agent flexibly sets the target permission to the specified state through system API or precise simulation of UI operations.

[0090] After the permission control agent completes the configuration, it quickly notifies the permission control agent itself to perform verification.

[0091] The permission control agent quickly reads the permission status of the operating system to verify whether the pre-configuration is successful.

[0092] If the verification fails, the permission control agent immediately reports the error to the task planning agent, which then retries or adjusts the preconfigured strategy.

[0093] If the verification is successful, the authority control agent reports to the task planning agent that the preset environment is ready.

[0094] It saves time and is less prone to errors. In continuous testing scenarios (such as permission status switching stress testing), it can achieve high-frequency, long-cycle automated execution. The test efficiency is high and can meet the testing needs of rapid iteration of batch devices.

[0095] In one embodiment, the step S4 further includes, if a non-standard permission prompt is detected, using image recognition and natural language processing technology to understand the prompt content and make a selection.

[0096] Specifically, if a non-standard or special permission prompt is detected, the request-response agent uses image recognition and natural language processing technology to deeply understand the prompt content and make accurate choices.

[0097] Status inspection and data collection

[0098] The permission control agent continuously and comprehensively performs the following operations:

[0099] At set intervals, the uiautomator dump command is executed regularly to obtain UI information, and the screen image at the corresponding moment is captured through adb screenshot. The complete UI status data is constructed by combining the two information.

[0100] Use XML parsing algorithms to deeply parse the obtained XML layout file and extract permission-related data, such as the display status, location coordinates, button text, etc. of the permission pop-up window.

[0101] Use a large language model to intelligently parse UI elements and continuously monitor changes in permission status, covering the initial state, intermediate states during the permission request process, and the final state after the operation is completed, and record the time nodes of status changes in detail.

[0102] Focus on filtering and recording key information:

[0103] Completely record the detailed content and exact coordinates of the permission pop-up window to analyze the rationality of the user interaction interface.

[0104] Real-time monitoring of device interface status changes, such as the enable and disable status of hardware interfaces such as cameras and microphones.

[0105] Detailed records of the application's response behavior during permission operations, including interface jumps, prompt information display, etc.

[0106] In one embodiment, the S4 further includes recording the operation execution process and application response of the operating system of the device under test.

[0107] The request-response agent monitors and processes the following operations in real time:

[0108] Continuously monitor permission request events initiated by applications to ensure immediate response to permission requests.

[0109] Based on the currently configured permission status, respond intelligently and accurately:

[0110] For authorized permission requests, a normal access permission response is given to simulate the system behavior after the user agrees to the authorization.

[0111] For unauthorized permission requests, an access denied prompt is returned to prevent unauthorized access.

[0112] When faced with a request for restricted permissions, an access prompt that meets the restriction conditions is returned, reflecting the system's control strategy for restricted permissions.

[0113] Use the adb tap command to simulate user click operations, respond to permission pop-up windows in a variety of ways, and simulate different permission interaction behaviors of users in actual use, such as confirming authorization, rejecting authorization, selecting this time only, etc., and record the operation time and results of each response in detail.

[0114] Beneficial effects:

[0115] Use the adb logcat command to obtain detailed call logs of the test application during the permission request and response process, including system-level logs and application-customized logs, to provide comprehensive data support for subsequent analysis.

[0116] Relying on the decision-making analysis capabilities of large language models, it deeply analyzes the permission "request-response" interaction pattern, accurately identifies abnormal response behaviors, such as unreasonable permission request frequency, abnormal response delays, etc., and classifies and marks abnormal behaviors.

[0117] Based on the generated test tasks, a structured task list is generated, the dependencies between tasks are sorted out, and detailed execution conditions are set to ensure the orderly and efficient execution of test tasks.

[0118] After the use case execution is completed, the task planning agent notifies the authority control agent to perform post-operation status verification.

[0119] The permission control agent reads the current permission status of the system and compares it with the expected status in detail.

[0120] The permission control agent also verifies the availability status of relevant system functions (such as camera, microphone, location services, etc.).

[0121] The permission control agent records the changes in permission status in detail, including the change time, change content, associated status, etc.

[0122] The authority control agent reports the verification results to the task planning agent.

[0123] In one embodiment, the test conclusion in S6 includes: pass or fail, and an exception description.

[0124] Specifically, the task planning agent passes the use case execution results and the status check results of the authority control agent to the result analysis agent.

[0125] Result Analysis The agent conducts a comprehensive and in-depth analysis of the test results based on the predefined Android permission control standard specifications.

[0126] Result Analysis The intelligent agent accurately identifies abnormal phenomena during the test process, such as inconsistent permission status, out-of-bounds access, silent authorization, etc.

[0127] The result analysis agent generates detailed test conclusions based on the analysis results, including pass / fail judgment, exception description, severity assessment, etc.

[0128] The result analysis agent returns the test conclusions to the task planning agent.

[0129] Furthermore, the result analysis agent performs the following operations:

[0130] Data structured processing:

[0131] Accurately record the timing information of permission operations in chronological order to form a complete operation chain.

[0132] Describe in detail the trajectory of permission status changes and show the evolution from the initial state to the final state.

[0133] Abnormal events are clearly marked to facilitate subsequent rapid location and analysis.

[0134] Abnormal behavior analysis:

[0135] Carefully compare the expected and actual results of different permission status changes to accurately determine whether there is a permission status error, that is, the actual status does not match the expected status.

[0136] In-depth inspection of function execution to determine whether there are any functional anomalies, such as if the function does not execute normally as expected after permission is granted.

[0137] Strictly identify unauthorized behavior and detect whether there are violations where unauthorized functions are available.

[0138] Risk Assessment Report:

[0139] Conduct detailed statistics on the types of defects found and analyze the frequency and distribution of each type of defect.

[0140] Assess the risk level based on the severity and impact of the defect, which will serve as a reference for subsequent testing.

[0141] Based on the problems discovered, we combine the system architecture and business logic to generate targeted optimization suggestions, including permission configuration adjustments, code repair directions, etc.

[0142] If anomalies are found during the analysis process, we will conduct an in-depth analysis of the cause of the problem, assess the scope of the problem's impact on system security and user experience, and propose specific repair plans and improvement measures.

[0143] The task planning agent receives the test conclusions and updates the execution status of the test task tree in a timely manner.

[0144] If anomalies or potential risks are detected, the task planning agent automatically and flexibly adjusts the testing strategy:

[0145] In-depth testing of specific permission paths significantly increases the complexity of test cases.

[0146] Significantly expand the testing breadth of suspected defect areas to fully cover relevant permission combination scenarios.

[0147] The task planning agent wisely selects the next test task to continue executing based on the adjusted strategy.

[0148] Repeat the tasks preset in the test task tree until all test tasks are completed or the termination condition is met.

[0149] In one embodiment, the S7 also includes automatically adjusting the test strategy if an anomaly or potential risk is detected, increasing the depth and breadth of the test, summarizing all test conclusions and automatically generating an optimization plan, and proposing repair suggestions.

[0150] The task planning agent receives the test conclusions and updates the execution status of the test task tree in a timely manner.

[0151] If anomalies or potential risks are detected, the task planning agent automatically and flexibly adjusts the testing strategy:

[0152] In-depth testing of specific permission paths significantly increases the complexity of test cases.

[0153] Significantly expand the testing breadth of suspected defect areas to fully cover relevant permission combination scenarios.

[0154] The task planning agent wisely selects the next test task to continue executing based on the adjusted strategy.

[0155] Repeat the test tasks in the task tree until all test tasks are completed or the termination condition is met.

[0156] After all test tasks are completed, the task planning agent will summarize all test results in the system.

[0157] The task planning agent calls the result analysis agent to generate a comprehensive test report with rich content.

[0158] The test report includes the overall assessment results of the permission control mechanism, details of each permission test, a list of discovered security defects and their severity, and improvement suggestions.

[0159] Based on the test results, the system automatically generates a scientific and reasonable optimization plan for the permission control mechanism and proposes specific and feasible repair suggestions for the problems found.

[0160] Output the test report and improvement suggestions to complete the entire testing process.

[0161] Through the implementation of the above steps, this method realizes the fully automated and standardized testing of the Android permission control mechanism, overcomes the limitations of the traditional manual testing mode, improves the testing efficiency and accuracy, and provides effective technical support for the security assessment of Android terminal devices.

[0162] Through the above-mentioned automated evaluation method based on multi-agent collaboration driven by a large language model, it is possible to achieve efficient, comprehensive, and intelligent evaluation of the Android permission control mechanism, significantly reducing reliance on human expert experience, improving test depth and accuracy, providing strong support for the security optimization of Android terminal devices, and effectively ensuring user data security and privacy protection.

[0163] See also Figure 3 , Figure 3 This is a schematic diagram of an open source large language model of an automated evaluation system for an Android permission control mechanism described in the present invention interacting with various intelligent agents and driving the test process.

[0164] The technical solution to achieve the second purpose of the present invention is: to adopt an intelligent testing system for the entire process of Android permission control function security assessment based on a large language model (LLM) to drive the collaboration of multiple agents, to build a complete Android permission control function security assessment workflow, including the collaborative work of five professional agents: task planning agent, permission control agent, use case execution agent, request response agent, and result analysis agent, to automatically perform the full-link testing process from permission request interaction to call result analysis.

[0165] An automated evaluation system for Android permission control mechanisms, including:

[0166] Startup module: used to initialize the test environment and establish communication with the device under test;

[0167] Task planning agent: used to scan the operating system permission list of the device under test, obtain the permission information of the device under test, build a permission detection list, build multi-dimensional test tasks, preset test strategies, generate a test task tree, and start the test process;

[0168] The permission control agent is used to read the current permission status of the operating system, set the target permission to be tested to a specified status, and pre-configure the target permission to be tested;

[0169] Use case execution agent: used to trigger the target permission behavior to be tested, monitor the UI changes of the operating system in real time, detect permission request pop-ups, and automatically select the corresponding option according to the preset test strategy when a permission request prompt is detected. If a non-standard permission prompt is detected, image recognition and natural language processing technology are used to understand the prompt content and make a selection;

[0170] Request-response agent: used to read the current permission status of the operating system, perform post-operation status verification, compare it with the expected permission status, verify the availability status of the operating system function and record changes in the permission status;

[0171] Result Analysis Agent: used to analyze test results, identify abnormal situations during the test process, and generate test conclusions;

[0172] Evaluation module: used to update the execution status of the test task tree, select the next test task to continue execution until all test tasks are completed, and output a test report.

[0173] Specifically, the functions of the five intelligent agents also include:

[0174] (1) The task planning agent, as the core task planning of the entire automated test system, is responsible for formulating test strategies and coordinating the work of other agents. It builds a permission checklist based on device permission information, generates a test task tree, generates a test task sequence according to the tree structure path, schedules and coordinates other agents to execute test tasks according to tasks, and receives test results to determine the next test direction.

[0175] (2) The permission control agent receives the task requirements issued by the task planning agent, performs specific permission configuration and status check, pre-configures the status of the target permission according to the task requirements, checks the changes in the permission status before and after the test, verifies whether the permission settings are effective as expected, and monitors the relevant UI status (such as Bluetooth switch, location service status).

[0176] (3) The use case execution agent receives the task requirements issued by the task planning agent to execute specific test cases, perform specific permission-related operations (such as calling the camera, accessing contacts), trigger permission requests or permission usage behaviors, and record the execution process and application response.

[0177] (4) The request-response agent receives the system permission request prompt issued by the task planning agent, configures the permission status through the system interface or UI operation, simulates user operations to grant, deny or revoke permissions, and handles different styles of permission dialog boxes and prompts as well as possible secondary confirmations or special prompts. If a non-standard or special permission prompt is detected, the request-response agent uses image recognition and natural language processing technology to deeply understand the prompt content and make accurate choices, including status inspection and data collection, key filtering and recording of key information.

[0178] (5) Result analysis agent: collects test data fed back by each agent, compares the actual results with the expected behavior, conducts compliance assessment based on the test standards, and generates test conclusions and problem reports.

[0179] The large language model-driven intelligent agent system is implemented as follows:

[0180] This invention focuses on the efficient application of existing open-source large models and the innovative architecture of intelligent agent systems, and uses open-source models to build a flexible and efficient intelligent agent collaboration system. Open-source large language models such as Deepseek and Qwen are used as the core inference engines of the intelligent agents, and dedicated prompt templates are designed for five types of intelligent agents respectively, including task descriptions, domain knowledge, constraints, and output formats. An example of the prompt structure for the task planning intelligent agent is as follows:

[0181] {Role definition}: Task planning expert for the Android permission testing system

[0182] {Background knowledge}: <Knowledge related to the Android permission system>

[0183] {Current status}: <Device information, system version, tested tasks>

[0184] {Task objective}: Plan the next test task based on the current status

[0185] {Output format}: Output strictly in JSON format, including fields such as task number, description, execution conditions, etc.

[0186] The intelligent agent communication mechanism is as follows:

[0187] Intelligent agents communicate through standardized JSON messages, and the messages contain the following fields:

[0188] sender: Identifier of the sending intelligent agent

[0189] receiver: Identifier of the receiving intelligent agent

[0190] message_type: Message type (instruction / return / notification)

[0191] task_id: Associated task identifier

[0192] content: Message content

[0193] timestamp: Timestamp

[0194] The collaboration process is driven by the task planning intelligent agent. Based on the event trigger mechanism, each intelligent agent reports the result to the task planning intelligent agent after completing the task, and the task planning intelligent agent initiates subsequent tasks accordingly.

[0195] An automated evaluation device for an Android permission control mechanism includes a processor and a memory. The memory stores a computer program, and the computer program can be executed by the processor to implement the method described in any of the above embodiments.

[0196] Specifically, the computer is not limited to one type, for example, a mobile phone, a computer, or other microcomputers. Utilizing the automated evaluation method and system, an automated evaluation of the Android system's permission control mechanism is implemented, achieving test standardization, ensuring test completeness, and improving test efficiency.

[0197] (1) The automated evaluation method for the Android permission control mechanism provided by the present invention uses an intelligent testing system based on a large language model-driven multi-agent collaboration to achieve automation and intelligence in the entire process of security evaluation of the Android permission control function, reducing reliance on expert experience and alleviating the burden on testers. The large language model-driven intelligent agent collaboration model makes the testing process more automated and intelligent. The links that rely on human expert experience for judgment and planning in the existing technology can now be efficiently completed by intelligent agents with the help of model capabilities. From test task planning to abnormal behavior identification and analysis, omissions and deviations caused by human subjective factors are reduced, and the objectivity and stability of the test are improved.

[0198] (2) The present invention arranges and decomposes tasks from both depth and breadth through the constructed permission tree, covering single permissions, permission combinations, multiple permission types and application scenarios. It also designs combined tasks for permission groups, fully considering various relationships and impacts to ensure comprehensive testing.

[0199] (3) The present invention achieves automation by constructing multiple agents to work together. The use case execution agent can quickly and accurately activate applications, configure permission status, and trigger permission use case calls based on the task list, while recording key parameters and timestamps. The request response agent monitors permission request events in real time and responds quickly, while obtaining detailed call logs. The permission control agent obtains UI information and captures screen images at set time intervals, efficiently collecting data. The agents work together closely, greatly shortening the test cycle and improving test efficiency.

[0200] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Under the concept of the present invention, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes in different aspects of the present invention as described above. For the sake of simplicity, they are not provided in detail. Although the present invention is described in detail with reference to the above embodiments, ordinary technicians in this field should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An automated evaluation method for Android permission control mechanism, characterized in that: include: S1: Initialize the test environment and establish communication with the device under test; S2: Scan the operating system permission list of the device under test, obtain the permission information of the device under test, build a permission detection list, build a multi-dimensional test task, preset a test strategy, generate a test task tree, and start the test process; S3: Read the current permission status of the operating system, set the target permission to be tested to a specified status, and pre-configure the target permission to be tested; S4: Triggering the target permission behavior to be tested, monitoring the UI changes of the operating system in real time, detecting permission request pop-ups, and automatically selecting the corresponding option according to the preset test strategy when a permission request prompt is detected. If a non-standard permission prompt is detected, using image recognition and natural language processing technology to understand the prompt content and make a selection; S5: Read the current permission status of the operating system, perform post-operation status verification, compare it with the expected permission status, verify the availability status of the operating system function and record changes in the permission status; S6: Analyze test results, identify abnormal conditions during the test, and generate test conclusions; S7: Update the execution status of the test task tree, select the next test task to continue execution, until all test tasks are completed, and output a test report.

2. The automated evaluation method for an Android permission control mechanism according to claim 1, characterized in that: The obtaining of the permission information of the device under test in S2 further includes collecting system-level and application-level permissions.

3. The automated evaluation method for an Android permission control mechanism according to claim 1, characterized in that: The multi-dimensional testing task constructed in S2 includes generating a permission type dimension, an authorization status dimension, an operation scenario dimension, and a permission dependency dimension.

4. The automated evaluation method for an Android permission control mechanism according to claim 1, wherein: Reading the current permission status of the operating system in S3 also includes verifying whether the pre-configuration of the target permission to be tested is successful. If the verification fails, an error report is sent to retry or adjust the pre-configuration strategy.

5. The automated evaluation method for an Android permission control mechanism according to claim 1, characterized in that: The step S4 further includes recording the operation execution process and application response of the operating system of the device under test.

6. The automated evaluation method for an Android permission control mechanism according to claim 1, characterized in that: The test conclusion in S6 includes: pass or fail, and abnormality description.

7. The automated evaluation method for an Android permission control mechanism according to claim 1, characterized in that: The S7 also includes automatically adjusting the test strategy if an anomaly or potential risk is detected, increasing the depth and breadth of the test, summarizing all test conclusions and automatically generating an optimization plan, and proposing repair suggestions.

8. An automated evaluation system for Android permission control mechanisms, characterized in that: include: Startup module: used to initialize the test environment and establish communication with the device under test; Task planning agent: used to scan the operating system permission list of the device under test, obtain the permission information of the device under test, build a permission detection list, build multi-dimensional test tasks, preset test strategies, generate a test task tree, and start the test process; The permission control agent is used to read the current permission status of the operating system, set the target permission to be tested to a specified status, and pre-configure the target permission to be tested; Use case execution agent: used to trigger the target permission behavior to be tested, monitor the UI changes of the operating system in real time, detect permission request pop-ups, and automatically select the corresponding option according to the preset test strategy when a permission request prompt is detected. If a non-standard permission prompt is detected, image recognition and natural language processing technology are used to understand the prompt content and make a selection; Request-response agent: used to read the current permission status of the operating system, perform post-operation status verification, compare it with the expected permission status, verify the availability status of the operating system function and record changes in the permission status; Result Analysis Agent: used to analyze test results, identify abnormal situations during the test process, and generate test conclusions; Evaluation module: used to update the execution status of the test task tree, select the next test task to continue execution until all test tasks are completed, and output a test report.

9. An automated evaluation device for an Android permission control mechanism, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and is characterized in that the computer program can be executed by the processor to implement the method according to any one of claims 1 to 7.

Images