Automatic evaluation method, system and device for Android authority control mechanism

Abstract

The invention discloses an automatic evaluation method for an Android authority control mechanism, which comprises the following steps: S1, initializing a test environment, and establishing communication with to-be-tested equipment; s2, scanning an operating system permission list of the to-be-tested equipment, generating a test task tree, and starting a test process; s3, pre-configuring the target permission to be tested; s4, triggering the target to-be-tested permission behavior; s5, reading the current permission state of the target system, and performing state verification after operation; s6, analyzing a test result, and generating a test conclusion; and S7, updating the execution state of the test task tree, selecting the next test task to continue execution until all the test tasks are completed, and outputting a test report. Automation of the test process is realized, and boundary scenes are fully covered to ensure test completeness; the time is saved, errors are not prone to occurring, the testing efficiency is high, the testing requirement for rapid iteration of batch equipment can be met, structured storage can be achieved, intelligent comparison with expected rules can be achieved, and the root cause analysis efficiency is high.

Description

Technical Field

[0001] The present invention belongs to the field of Android system access control, and particularly relates to an automated evaluation method, system and device for an Android permission control mechanism. Background Art

[0002] In the field of evaluating the security capabilities of Android operating system control mechanisms, the current common method is to use the "permission tool + manual testing" mode for detection. In this mode, a permission interface testing tool is manually called to trigger permission request use cases, and the user is simulated to manage application permissions in the terminal device system settings, including operations such as granting, revoking, or restricting permission usage. Then, it is manually evaluated whether the control mechanism of the permission control mechanism for the application meets the expectations and whether there are defects. This mode has the following four main deficiencies: I. Strong dependence on manual experience and low test standardization: There are a wide variety of customizations of the Android operating system firmware, and fragmentation is serious. There are differences between the user interface (UI) and the operating system version, and it is also in frequent updates and changes. The existing technology relies on testers to manually configure permission states, execute test cases, and manually judge the results, requiring testers to deeply master the permission management rules and UI characteristics of each version of the system. However, manual operations are prone to problems such as incorrect permission configuration, omission of test case execution, and subjective deviation in result judgment due to experience differences. The test process is difficult to standardize, and the reproducibility of test results is poor.

[0003] II. Lack of dynamic adaptation ability in the test process and limited coverage scenarios: The existing "permission tool + manual testing" mode requires pre-written static test cases and cannot dynamically adjust the test strategy according to the characteristics of the system version under test. For example, there are differences in the runtime request mechanisms for dangerous permissions (such as dynamic pop-ups and floating window permissions) in different Android versions. Manual testing is difficult to quickly adapt to the changes in the permission management logic of multiple versions, resulting in insufficient coverage of boundary scenarios (such as multiple grants / revocations of permissions and combination permission conflicts), and it is difficult to ensure test completeness.

[0004] III. Lack of automated and intelligent evaluation and low manual testing efficiency: For the test requirements of permission control mechanisms for multiple models and multiple system versions, a large number of operations such as permission configuration, application installation, and use case triggering need to be repeated. The existing technology relies on manual item-by-item operations, which are time-consuming and error-prone. Especially in continuous test scenarios (such as stress testing of permission state switching), high-frequency and long-cycle automated execution cannot be achieved, resulting in low test efficiency and difficulty in meeting the test requirements for the rapid iteration of batch devices.

[0005] IV. Insufficient ability to record and analyze test results: Manual testing relies on subjective recording of permission control behaviors (such as pop-up window content and permission status synchronization delay), and lacks automated monitoring means for abnormal events (such as unauthorized permission calls and background silent authorization). In addition, test logs are mostly discrete text records, which are difficult to store structurally and intelligently compare with expected rules, resulting in low efficiency in defect location and root cause analysis. Summary of the Invention

[0006] Based on this, it is necessary to propose an automated evaluation method for the Android permission control mechanism to address the problems of low standardization of the above tests, limited coverage scenarios, low efficiency of manual testing, and insufficient ability to record and analyze test results.

[0007] To achieve the above object, the present invention adopts the following technical solutions: An automated evaluation method for an Android permission control mechanism, comprising: S1: Initialize the test environment and establish communication with the device under test; S2: Scan the operating system permission list of the device under test, obtain the permission information of the device under test, construct a permission detection list, construct multi-dimensional test tasks, preset test strategies, generate a test task tree, and start the test process; S3: Read the current permission status of the operating system, set the target permission to be tested to a specified status, and pre-configure the target permission to be tested; S4: Trigger the behavior of the target permission to be tested, monitor the UI changes of the operating system in real time, detect permission request pop-ups, and automatically select the corresponding option according to the preset test strategy when a permission request prompt is detected; S5: Read the current permission status of the target system, verify the status after the operation, compare it with the expected permission status, verify the availability status of the operating system function, and record the change of the permission status; S6: Analyze the test results, identify abnormal situations during the test, and generate a test conclusion; S7: Update the execution status of the test task tree, select the next test task to continue execution until all test tasks are completed, and output a test report.

[0008] In some embodiments, the obtaining of the permission information of the device under test in S2 further includes collecting system-level and application-level permissions.

[0009] In some embodiments, the constructing of the multi-dimensional test tasks in S2 includes generating dimensions of permission type, authorization status, operation scenario, and permission dependency.

[0010] In some embodiments, reading the current permission status of the operating system in S3 further includes verifying whether the pre-configuration of the target permission to be tested is successful. If the verification fails, an error report is sent, and a retry or adjustment of the pre-configuration policy is performed.

[0011] In some embodiments, S4 further includes, if a non-standard permission prompt is detected, using image recognition and natural language processing technologies to understand the prompt content and make a selection.

[0012] In some embodiments, S4 further includes recording the operation execution process of the operating system of the device under test and the application response situation.

[0013] In some embodiments, the test conclusion in S6 includes: passed or determined to fail, and an exception description.

[0014] In some embodiments, S7 further includes, if an exception or potential risk is detected, automatically adjusting the test strategy, increasing the depth and breadth of the test, summarizing all test conclusions and automatically generating an optimization plan, and proposing repair suggestions.

[0015] An automated evaluation system for an Android permission control mechanism, comprising: A startup module: used to initialize the test environment and establish communication with the device under test; A task planning agent: used to scan the operating system permission list of the device under test, obtain the permission information of the device under test, construct a permission detection list, construct a multi-dimensional test task, preset a test strategy, generate a test task tree, and start the test process; A permission control agent: used to read the current permission status of the operating system, set the target permission to be tested to a specified state, and pre-configure the target permission to be tested; A use case execution agent: used to trigger the behavior of the target permission to be tested, monitor the UI changes of the operating system in real time, detect permission request pop-ups, and when a permission request prompt is detected, automatically select the corresponding option according to the preset test strategy; A request response agent: used to read the current permission status of the target system, verify the post-operation status, compare it with the expected permission status, verify the availability status of the operating system function, and record the change of the permission status; A result analysis agent: used to analyze the test results, identify abnormal situations in the test process, and generate test conclusions; An evaluation module: used to update the execution status of the test task tree, select the next test task to continue execution until all test tasks are completed, and output a test report.

[0016] An automated evaluation device for an Android permission control mechanism, comprising a processor and a memory, wherein a computer program is stored in the memory and can be executed by the processor to implement the method described in any of the above embodiments.

[0017] Compared with the currently widely used "permission tool + manual testing" mode, the present invention has the following advantages through automated evaluation: 1. The testing process is automated, and the reproducibility of the test results is good. 2. It can quickly adapt to the changes in the permission management logic of multiple versions, fully covering boundary scenarios (such as multiple grants / revocations of permissions, combination permission conflicts), ensuring the completeness of testing. 3. It saves time and is not prone to errors. In continuous testing scenarios (such as stress testing of permission state switching), it can achieve high-frequency and long-cycle automated execution, with high testing efficiency, and can meet the testing requirements for the rapid iteration of batch devices. 4. It can store structured data and perform intelligent comparison with expected rules, with high efficiency in defect location and root cause analysis. Brief Description of the Drawings

[0018] Wherein: Figure 1 is a flowchart of an automated evaluation method for an Android permission control mechanism according to the present invention; Figure 2 is a schematic diagram of the architecture of a multi-agent collaboration system driven by a large language model for an automated evaluation system of an Android permission control mechanism according to the present invention.

[0019] Figure 3 is a schematic diagram of an open-source large language model of an automated evaluation system of an Android permission control mechanism according to the present invention interacting with each agent and driving the test process. Detailed Embodiments

[0020] To facilitate the understanding of the present invention, the present invention will be described in more detail below with reference to the accompanying drawings and specific embodiments. It should be noted that when an element is expressed as "connected" to another element, it can be directly on the other element, or there can be one or more intermediate elements therebetween. The terms "upper", "lower", "left", "right", "upper end", "lower end", "top" and "bottom" etc. used in this specification indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus cannot be construed as a limitation to the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes and cannot be construed as indicating or implying relative importance.

[0021] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by those skilled in the technical field to which this invention belongs. The terms used in the specification of this invention are for the purpose of describing specific embodiments only and are not intended to limit the invention.

[0022] To solve the problems of low test standardization, lack of dynamic adaptation ability in the test process, limited coverage scenarios, lack of automated and intelligent evaluation, low efficiency of manual testing, and insufficient ability to record and analyze test results in the prior art, the following will be combined with the attached Figures 1 to 3 A method, system, and device for automated evaluation of an Android permission control mechanism provided by an embodiment of the present invention will be described in detail.

[0023] The technical solution for achieving the first object of the present invention is an automated evaluation method for an Android permission control mechanism. Please refer to Figure 1 and Figure 2 , Figure 1 is a flowchart of an automated evaluation method for an Android permission control mechanism according to the present invention; Figure 2 is a schematic diagram of the architecture of a multi-agent collaboration system driven by a large language model for an automated evaluation system of an Android permission control mechanism according to the present invention.

[0024] An automated evaluation method for an Android permission control mechanism includes: S1: Initialize the test environment and establish communication with the device under test; S2: Scan the operating system permission list of the device under test, obtain the permission information of the device under test, construct a permission detection list, construct multi-dimensional test tasks, preset test strategies, generate a test task tree, and start the test process; S3: Read the current permission status of the operating system, set the target permission to be tested to a specified status, and pre-configure the target permission to be tested; S4: Trigger the behavior of the target permission to be tested, monitor the UI changes of the operating system in real time, detect permission request pop-ups, and automatically select the corresponding option according to the preset test strategy when a permission request prompt is detected; S5: Read the current permission status of the target system, verify the status after the operation, compare it with the expected permission status, verify the availability status of the operating system function, and record the change of the permission status; S6: Analyze the test results, identify abnormal situations in the test process, and generate a test conclusion; S7: Update the execution status of the test task tree, select the next test task to continue execution until all test tasks are completed, and output a test report.

[0025] Specifically, S1: Establish a stable connection with the Android device under test through ADB (Android Debug Bridge) to ensure normal communication and correct device recognition; S2: The task planning agent comprehensively scans the operating system version and permission management mechanism characteristics of the device under test, details the system compatibility and special permission management rules, accurately collects the list of permission types supported by the device, covering general permissions, dangerous permissions, special permissions, etc., and marks the usage frequency and sensitivity of the permissions to build a more complete structured permission detection list. In addition to recording the default status, grouping situation and related APIs of each permission, the dependency relationship and potential impact of the permissions are also supplemented. According to the permission detection list, a multi-dimensional test task tree is generated to ensure comprehensive test coverage; The task planning agent selects specific test tasks according to the test task tree, officially starts the test process, and sends permission pre-configuration instructions to the permission control agent; S3: The permission control agent sets the target permission to the specified status through the API of the operating system of the device under test or simulates UI operations. After the permission control agent completes the configuration, it performs self-verification and reads the permission status of the operating system to verify whether the pre-configuration is successful; When the verification is successful, the permission control agent reports to the task planning agent that the pre-set environment is ready; S4: ① The task planning agent sends a test instruction to the test case execution agent. After receiving the test instruction, the test case execution agent calls the corresponding API or performs specific operations to trigger the relevant behaviors of the permissions; ② At the same time, the task planning agent activates the monitoring mode of the request response agent. After being activated, the request response agent monitors the system UI changes in real time, detects permission request pop-ups. When a permission request is detected, the request response agent automatically selects the corresponding option according to the preset policy. The options include allow, deny, allow only this time, etc.; S5: After the test case execution agent completes the relevant behaviors of the permissions, the task planning agent notifies the permission control agent to verify the status after the operation. The permission control agent reads the current permission status of the operating system of the device under test and compares it with the expected status; S6: Analyze the test results to generate a test conclusion. The test includes pass, fail determination, exception description, severity assessment, etc.

[0026] S7: After completing the test, update the execution status of the test task tree in a timely manner, adjust the test strategy, and select the next test task until all test tasks are completed or the termination condition is met.

[0027] Based on the test results, automatically generate an optimized plan for the permission control mechanism, put forward specific and feasible repair suggestions for the discovered problems, output the test report and improvement suggestions, and complete the entire test process.

[0028] The present invention has the following advantages through automated evaluation compared with the currently widely used "permission tool + manual testing" mode: 1. The testing process is automated, and the test results have good reproducibility. 2. It can quickly adapt to changes in the permission management logic of multiple versions, fully cover boundary scenarios (such as multiple permission grants / revocations, combined permission conflicts), and ensure test completeness. 3. It saves time and is not prone to errors. In continuous testing scenarios (such as stress testing for permission status switching), it can achieve high-frequency and long-cycle automated execution, with high test efficiency, and can meet the test requirements for the rapid iteration of batch devices. 4. It can store structured data and perform intelligent comparison with expected rules, and has high efficiency in defect location and root cause analysis.

[0029] In one embodiment, obtaining the permission information of the device under test in S2 further includes collecting system-level and application-level permissions.

[0030] Specifically, establish a stable connection with the Android device under test through ADB (Android Debug Bridge) to ensure normal communication and correct device identification. The task planning agent sequentially performs the following core operations: With the help of the adb shell pm list permissions command, obtain the complete list of permissions supported by the device under test, covering various system-level and application-level permissions. Use the adb shell dumpsys package command to deeply analyze and record the initial state of permissions, including default grants, default denials, and related permission attributes.

[0031] Construct a fine-grained permission tree structure to clearly show the hierarchical relationship of permissions, clarify the permission group information to which each permission belongs, and provide an intuitive basis for subsequent task planning. Utilize the reasoning ability of the large language model to intelligently generate comprehensive and detailed test tasks from multiple dimensions based on the constructed permission tree.

[0032] The test case execution agent performs the following operations: a. Use the start_activity function to activate the application under test to ensure that the application under test is in a testable state and the running environment meets the test requirements. b. Deeply analyze the structured task list generated by the task planning agent to clarify the specific operation steps and objectives of each test task.

[0033] c. Use the adb shell pm grant / revoke command to dynamically and precisely configure the permission status according to the test task requirements, simulating the actual scenarios of different permission grants and revocations. d. Send a preset broadcast notification through adb shell am broadcast to trigger the corresponding permission use case calls of the test application, ensuring that the permission use cases are correctly executed in the set appropriate scenarios, and recording the key parameters and timestamps during the call process. The testing process is automated, and the test results have good reproducibility. It can quickly adapt to the changes in the multi-version permission management logic, fully covering the boundary scenarios (such as multiple permission grants / revocations, combined permission conflicts), ensuring the test completeness.

[0034] In one embodiment, the constructing of the multi-dimensional test task in S2 includes generating dimensions of permission type, authorization status, operation scenario, and permission dependency.

[0035] Specifically, the test task tree adopts a multi-dimensional design, and the construction algorithm is as follows: The root node is set as the Android permission test of the device under test The first-level branches are divided by permission type (normal permissions, dangerous permissions, special permissions) The second-level branches are divided by permission group (location, storage, camera, etc.) The third-level branches are divided by specific permissions (accurate location, approximate location, etc.) The fourth-level branches are divided by authorization status (not requested, authorized, denied, only this time, etc.) The fifth-level branches are divided by test scenario (foreground use, background use, frequent requests, etc.) Depth dimension: For a single permission, design basic test tasks such as grant and revoke respectively, and conduct in-depth multi-level permission combination tests, such as nested combination tests of permissions with different danger levels; at the same time, set boundary scenario tests to simulate multiple authorization and revocation operations to test the system's handling ability of complex permission changes. Breadth dimension: Comprehensively cover various permission types, and conduct tests on dangerous permissions (such as key permissions involving user privacy like location, camera, microphone, etc.), normal permissions (such as conventional permissions like network access, vibration, etc.), and special permissions (such as permissions with special functions like installing applications, system settings, etc.); and cover the permission usage tests in different foreground and background scenarios to ensure the effectiveness of the permission mechanism in different operating environments.

[0036] Permission group dimension: Carefully design collaborative test tasks within the permission group to verify the stability of the collaboration of each permission within the same permission group; conduct cross-group permission interaction tests to explore the interaction effects between different permission groups; at the same time, set up permission conflict tests to simulate possible permission conflict scenarios and detect the conflict handling strategy of the system.

[0037] The test process is automated, the test results have good reproducibility, can quickly adapt to the changes in the multi-version permission management logic, fully cover the boundary scenarios (such as multiple grants / revocations of permissions, combined permission conflicts), and ensure the test completeness.

[0038] In one embodiment, reading the operating system permission status in S3 further includes verifying whether the target permission pre-configuration is successful. If the verification fails, send a report error and perform a retry or adjust the pre-configuration strategy.

[0039] Specifically, the task planning agent carefully selects the preset permission test tasks according to the test task tree and officially starts the test process.

[0040] The task planning agent timely sends the permission pre-configuration instruction to the permission control agent.

[0041] The permission control agent flexibly sets the target permission to the specified state through the system API or precisely simulates the UI operation.

[0042] After the permission control agent completes the configuration, it quickly notifies the permission control agent itself to perform verification.

[0043] The permission control agent quickly reads the operating system permission status to verify whether the pre-configuration is successful.

[0044] If the verification fails, the permission control agent immediately reports an error to the task planning agent, and the task planning agent performs a retry or adjusts the pre-configuration strategy.

[0045] If the verification is successful, the permission control agent reports to the task planning agent that the preset environment is ready.

[0046] Save time and are not prone to errors. In continuous test scenarios (such as permission status switching stress tests), they can achieve high-frequency and long-cycle automated execution, with high test efficiency and can meet the test requirements for the rapid iteration of batch devices.

[0047] In one embodiment, S4 further includes that if a non-standard permission prompt is detected, image recognition and natural language processing technologies are used to understand the prompt content and make a selection.

[0048] Specifically, if a non-standard or special permission prompt is detected, the request response agent uses image recognition and natural language processing technologies to deeply understand the prompt content and make a precise selection.

[0049] Status Check and Data Collection The permission control agent continuously and comprehensively performs the following operations: At set time intervals, periodically execute the uiautomator dump command to obtain UI information, capture the screen image at the corresponding moment through adbscreenshot, and combine the two pieces of information to construct complete UI status data. Use the XML parsing algorithm to deeply parse the obtained XML layout file, and extract permission-related data, such as the display status, position coordinates, button text, etc. of the permission pop-up window.

[0050] Use the large language model to intelligently parse UI elements, continuously monitor the changes in the permission status, covering the initial state, intermediate states during the permission request process, and the final state after the operation is completed, and record the time nodes of the state changes in detail. Focus on filtering and recording key information: Completely record the detailed content and accurate coordinates of the permission pop-up window to facilitate the analysis of the rationality of the user interface.

[0051] Real-time monitor the changes in the device interface status, such as the enabled and disabled states of hardware interfaces such as cameras and microphones. Detailedly record the response behaviors of the application during the permission operation, including interface jumps, prompt message displays, etc. In one embodiment, the S4 further includes recording the operation execution process of the operating system of the device under test and the application response situation.

[0052] The request response agent monitors and processes the following operations in real time: Continuously listen for permission request events initiated by the application to ensure an immediate response to the permission request. Based on the currently configured permission status, perform intelligent and accurate response processing: For authorized permission requests, give a normal access permission response to simulate the system behavior after the user agrees to authorize. For unauthorized permission requests, return a deny access prompt to prevent unauthorized access to permissions. In the face of restricted permission requests, return an access prompt that meets the restriction conditions to reflect the system's control strategy for restricted permissions. Use the adb tap command to simulate the user's click operation, perform diverse responses to the permission pop-up window, simulate different permission interaction behaviors of the user in actual use, such as confirming authorization, denying authorization, selecting only for this use, etc., and record the operation time and result of each response in detail.

[0053] Beneficial Effects: Obtain the detailed call logs of the test application during the permission request and response process through the adb logcat command, including system-level logs and application-customized logs, to provide comprehensive data support for subsequent analysis. Relying on the decision-making analysis ability of the large language model, deeply analyze the interaction mode of permission "request-response", accurately identify abnormal response behaviors, such as unreasonable permission request frequencies, abnormal response delays, etc., and classify and mark abnormal behaviors. According to the generated test tasks, organize and generate a structured task list, sort out the dependencies between tasks, and set detailed execution conditions to ensure the orderliness and efficiency of test task execution.

[0054] After the use case execution is completed, the task planning agent notifies the permission control agent to verify the operation status.

[0055] The permission control agent reads the current system permission status and makes a detailed comparison with the expected status.

[0056] The permission control agent also verifies the availability status of relevant system functions (such as cameras, microphones, location services, etc.).

[0057] The permission control agent details the changes in the permission status, including the change time, change content, associated status, etc.

[0058] The permission control agent reports the verification results to the task planning agent.

[0059] In one embodiment, the test conclusion described in S6 includes: passed or judged failed, abnormal description.

[0060] Specifically, the task planning agent transfers the use case execution results and the status check results of the permission control agent to the result analysis agent.

[0061] The result analysis agent conducts a comprehensive and in-depth analysis of the test results based on the predefined Android permission control standard specifications.

[0062] The result analysis agent accurately identifies abnormal phenomena during the test process, such as inconsistent permission status, permission overstep access, silent authorization, etc.

[0063] The result analysis agent generates a detailed test conclusion based on the analysis results, including passed, failed judgment, abnormal description, severity assessment, etc.

[0064] The result analysis agent returns the test conclusion to the task planning agent.

[0065] Furthermore, the result analysis agent performs the following operations: Data structured processing: Accurately record the timing information of permission operations in chronological order to form a complete operation chain. Detailedly depict the trajectory of permission status changes, showing the evolution process from the initial state to the final state. Clearly mark abnormal events for easy subsequent quick positioning and analysis. Abnormal behavior analysis: Carefully compare the expected results and actual results of different permission status changes to accurately judge whether there are permission status errors, that is, the situation where the actual state does not match the expected state. Deeply check the function execution situation to determine whether there are function abnormalities, such as the function not executing properly as expected after permission is granted.

[0066] Strictly identify unauthorized actions and detect whether there are violations of unauthorized but functional availability.

[0067] Risk assessment report: Make a detailed statistics on the discovered defect types and analyze the frequency and distribution of various defects. Based on the severity and impact scope of the defects, evaluate the risk level for subsequent testing reference. For the discovered problems, combined with the system architecture and business logic, generate targeted optimization suggestions, including permission configuration adjustments, code repair directions, etc. If abnormalities are found during the analysis process, deeply analyze the causes of the problems, evaluate the impact scope of the problems on system security and user experience, and propose specific repair plans and improvement measures. The task planning agent receives the test conclusion and timely updates the execution status of the test task tree.

[0068] If abnormalities or potential risks are detected, the task planning agent automatically and flexibly adjusts the test strategy: Deeply test specific permission paths, significantly increasing the complexity of test cases.

[0069] Greatly expand the test breadth of the suspected defect area to comprehensively cover relevant permission combination scenarios.

[0070] The task planning agent wisely selects the next test task to continue execution according to the adjusted strategy.

[0071] Repeat the tasks preset in the test task tree until all test tasks are completed or the termination conditions are met.

[0072] In one embodiment, the S7 further includes automatically adjusting the test strategy if abnormalities or potential risks are detected, increasing the depth and breadth of the test, summarizing all test conclusions and automatically generating an optimization plan, and proposing repair suggestions.

[0073] The task planning agent receives the test conclusions and updates the execution status of the test task tree in a timely manner.

[0074] If an anomaly or potential risk is detected, the task planning agent automatically and flexibly adjusts the test strategy: Conduct in-depth testing on specific permission paths, significantly increasing the complexity of test cases.

[0075] Greatly expand the test breadth of the suspected defect area, comprehensively covering relevant permission combination scenarios.

[0076] Based on the adjusted strategy, the task planning agent wisely selects the next test task to continue execution.

[0077] Repeat the test tasks of the task tree until all test tasks are completed or the termination conditions are met.

[0078] After all test tasks are completed, the task planning agent systematically summarizes all test results.

[0079] The task planning agent invokes the result analysis agent to generate a comprehensive test report with rich content.

[0080] The test report includes the overall evaluation results of the permission control mechanism, the details of each permission test, the list and severity of discovered security defects, improvement suggestions, etc.

[0081] Based on the test results, the system automatically generates a scientific and reasonable optimization plan for the permission control mechanism, and puts forward specific and feasible repair suggestions for the discovered problems.

[0082] Output the test report and improvement suggestions to complete the entire test process.

[0083] By implementing the above steps, this method realizes the fully automated and standardized testing of the Android permission control mechanism, overcomes the limitations of the traditional manual testing mode, improves the test efficiency and accuracy, and provides an effective technical guarantee for the security assessment of Android terminal devices.

[0084] Through the above automated evaluation method based on the collaboration of multiple agents driven by the large language model, it is possible to achieve an efficient, comprehensive, and intelligent evaluation of the Android permission control mechanism, greatly reducing the dependence on manual expert experience, improving the test depth and accuracy, providing strong support for the security optimization of Android terminal devices, and effectively ensuring user data security and privacy protection. Please refer to Figure 3 , Figure 3 which is a schematic diagram of the interaction between the open-source large language model of an automated evaluation system for an Android permission control mechanism described in the present invention and each agent to drive the test process.

[0085] The technical solution for achieving the second object of the present invention is as follows: An intelligent testing system for the entire process of security evaluation of Android permission control functions driven by a large language model (LLM) to collaborate multiple agents is adopted, and a complete workflow for security evaluation of Android permission control functions is constructed, including the collaborative work of five professional agents: a task planning agent, a permission control agent, a use case execution agent, a request response agent, and a result analysis agent, and automatically perform a full-link testing process from permission request interaction to call result analysis.

[0086] An automated evaluation system for an Android permission control mechanism, comprising: A startup module: used to initialize the test environment and establish communication with the device under test; A task planning agent: used to scan the operating system permission list of the device under test, obtain the permission information of the device under test, construct a permission detection list, construct multi-dimensional test tasks, preset test strategies, generate a test task tree, and start the test process; A permission control agent: used to read the current permission status of the operating system, set the target permission to be tested to a specified status, and pre-configure the target permission to be tested; A use case execution agent: used to trigger the behavior of the target permission to be tested, monitor the UI changes of the operating system in real time, detect permission request pop-ups, and automatically select corresponding options according to the preset test strategy when a permission request prompt is detected; A request response agent: used to read the current permission status of the target system, verify the status after operation, compare it with the expected permission status, verify the availability status of the operating system function, and record the change of the permission status; A result analysis agent: used to analyze the test results, identify abnormal situations in the test process, and generate a test conclusion; An evaluation module: used to update the execution status of the test task tree, select the next test task to continue execution until all test tasks are completed, and output a test report.

[0087] Specifically, the functions of the five agents also include: (1) The task planning agent, as the core task planning of the entire automated testing system, is responsible for formulating test strategies and coordinating the work of other agents. Construct a permission detection list according to the device permission information, generate a test task tree, generate a test task sequence according to the tree structure path, schedule and coordinate other agents to execute test tasks according to the tasks, and receive test results to determine the next test direction.

[0088] (2)The permission control agent receives the task requirements sent by the task planning agent, executes specific permission configurations and status checks, pre-configures the status of the target permissions according to the task requirements, checks the changes in the permission status before and after testing, verifies whether the permission settings take effect as expected, and monitors the relevant UI status (such as Bluetooth switch, location service status).

[0089] (3)The test case execution agent receives the task requirements sent by the task planning agent and executes specific test cases, performs specific permission-related operations (such as calling the camera, accessing contacts), triggers permission requests or permission usage behaviors, and records the execution process and application response.

[0090] (4)The request response agent receives the system permission request prompts sent by the task planning agent, configures the permission status through system interfaces or UI operations, simulates user operations to grant, reject, or revoke permissions, and processes different styles of permission dialog boxes and prompts, as well as possible secondary confirmations or special prompts.

[0091] (5)The result analysis agent collects the test data fed back by each agent, compares the actual results with the expected behaviors, conducts compliance evaluations according to the test criteria, and generates test conclusions and problem reports.

[0092] The large language model-driven intelligent agent system is implemented as follows: This invention focuses on the efficient application of existing open-source large models and the innovative architecture of the intelligent agent system, and uses open-source models to build a flexible and efficient intelligent agent collaboration system. Open-source large language models such as Deepseek and Qwen are used as the core inference engines of the intelligent agents, and dedicated prompt templates are designed for the five types of intelligent agents respectively, including task descriptions, domain knowledge, constraints, and output formats. An example of the prompt structure for the task planning agent: {Role definition}: Task planning expert for the Android permission testing system {Background knowledge}: <Knowledge related to the Android permission system> {Current status}: <Device information, system version, tested tasks> {Task objective}: Plan the next test task based on the current status {Output format}: Strictly output in JSON format, including fields such as task number, description, execution conditions, etc.

[0093] The intelligent agent communication mechanism is as follows: The intelligent agents communicate through standardized JSON messages, and the messages contain the following fields: sender: The identifier of the sending agent receiver: The identifier of the receiving agent message_type: Message type (instruction / reply / notification) task_id: Associated task identifier content: Message content timestamp: Timestamp The collaboration process is driven by a task planning agent. Based on an event-triggered mechanism, each agent reports the result to the task planning agent after completing the task, and the task planning agent initiates subsequent tasks accordingly.

[0094] An automated evaluation device for an Android permission control mechanism, comprising a processor and a memory. A computer program is stored in the memory and can be executed by the processor to implement the method described in any one of the above embodiments.

[0095] Specifically, the computer is not limited to one type. For example, mobile phones, computers, and other microcomputers. By using the automated evaluation method and system, the automatic evaluation of the Android system permission control mechanism is realized, the standardization of the test is achieved, the completeness of the test is ensured, and the test efficiency is improved.

[0096] (1) The automated evaluation method for the Android permission control mechanism provided by the present invention adopts an intelligent testing system based on multi-agent collaboration driven by a large language model, realizing the automation and intelligence of the entire process of Android permission control function security evaluation, reducing the dependence on expert experience and alleviating the burden on testers. The intelligent agent collaboration mode driven by the large language model makes the testing process more automated and intelligent. The links that rely on manual expert experience for judgment and planning in the prior art can be efficiently completed by the agent with the help of the model's capabilities. From test task planning to abnormal behavior recognition and analysis, the omissions and deviations caused by human subjective factors are reduced, and the objectivity and stability of the test are improved.

[0097] (2) The present invention conducts task arrangement and decomposition from both depth and breadth through the constructed permission tree, covering single permissions, permission combinations, various permission types and application scenarios, and also designs combined tasks for permission groups, fully considering various relationships and influences to ensure comprehensive testing.

[0098] (3) The present invention realizes automation by constructing multiple agents to work collaboratively. The use case execution agent can quickly and accurately activate applications, configure permission states, and trigger permission use case calls according to the task list, while recording key parameters and timestamps; the request response agent listens for permission request events in real time and responds quickly, while obtaining detailed call logs; the permission control agent regularly obtains UI information and captures screen images at set time intervals to efficiently collect data. Each agent divides the work and cooperates closely, greatly shortening the test cycle and improving the test efficiency. The above embodiments are only used to illustrate the technical solutions of the present invention, rather than limiting it; under the idea of the present invention, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations in different aspects of the present invention as described above, and for the sake of brevity, they are not provided in detail; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An automated evaluation method for an Android permission control mechanism, characterized in that include: S1: Initialize the test environment and establish communication with the device under test; S2: Scan the operating system permission list of the device under test, obtain the permission information of the device under test, build a permission detection list, build a multi-dimensional test task, preset a test strategy, generate a test task tree, and start the test process; S3: Read the current permission status of the operating system, set the target permission to be tested to a specified status, and pre-configure the target permission to be tested; S4: triggering the target permission behavior to be tested, monitoring the UI changes of the operating system in real time, detecting permission request pop-up windows, and automatically selecting corresponding options according to the preset test strategy when a permission request prompt is detected; S5: reading the current permission status of the target system, performing post-operation status verification, comparing it with the expected permission status, verifying the availability status of the operating system function and recording changes in the permission status; S6: Analyze the test results, identify abnormal conditions during the test, and generate test conclusions; S7: updating the execution status of the test task tree, selecting the next test task to continue executing, until all test tasks are completed, and outputting a test report.

2. The automated evaluation method of an Android permission control mechanism according to claim 1, characterized in that The obtaining of permission information of the device under test in S2 further includes collecting system-level and application-level permissions.

3. An automated evaluation method for an Android permission control mechanism according to claim 1, characterized in that The construction of the multi-dimensional test task described in S2 includes generating a permission type dimension, an authorization status dimension, an operation scenario dimension, and a permission dependency dimension.

4. An automated evaluation method for an Android permission control mechanism according to claim 1, characterized in that Reading the current permission status of the operating system in S3 also includes verifying whether the pre-configuration of the target permission to be tested is successful. If the verification fails, an error report is sent to retry or adjust the pre-configuration strategy.

5. The automated evaluation method of an Android permission control mechanism according to claim 1, characterized in that, The S4 also includes, if a non-standard permission prompt is detected, using image recognition and natural language processing technology to understand the prompt content and make a selection.

6. The automated evaluation method for an Android permission control mechanism according to claim 1, characterized in that, The S4 also includes recording the operation execution process and application response status of the operating system of the device under test.

7. The automated evaluation method of an Android permission control mechanism according to claim 1, characterized in that, The test conclusion described in S6 includes: pass or fail, and abnormal description.

8. An automated evaluation method for an Android permission control mechanism according to claim 1, characterized in that, The S7 also includes automatically adjusting the test strategy if an abnormality or potential risk is detected, increasing the depth and breadth of the test, summarizing all test conclusions and automatically generating an optimization plan, and proposing repair suggestions.

9. An automated evaluation system for an Android permission control mechanism, characterized in that, include: Startup module: used to initialize the test environment and establish communication with the device under test; Task planning agent: used to scan the operating system permission list of the device under test, obtain the permission information of the device under test, build a permission detection list, build multi-dimensional test tasks, preset test strategies, generate a test task tree, and start the test process; Permission control agent: used to read the current permission status of the operating system, set the target permission to be tested to a specified status, and pre-configure the target permission to be tested; Use case execution agent: used to trigger the target permission behavior to be tested, monitor the UI changes of the operating system in real time, detect permission request pop-up windows, and automatically select the corresponding option according to the preset test strategy when a permission request prompt is detected; Request response agent: used to read the current permission status of the target system, verify the status after operation, compare it with the expected permission status, verify the availability status of the operating system functions, and record the changes in the permission status; Result analysis agent: used to analyze the test results, identify abnormal situations during the test process, and generate test conclusions; Evaluation module: used to update the execution status of the test task tree, select the next test task to continue execution until all test tasks are completed, and output a test report.

10. An automated evaluation device for an Android permission control mechanism, characterized in that, It includes a processor and a memory, and a computer program is stored in the memory. It is characterized in that the computer program can be executed by the processor to implement the method according to any one of claims 1-8.

Images