Terminal agent operation safety control method and device, equipment and medium

Abstract

The application relates to the technical field of artificial intelligence, and discloses a terminal intelligent agent operation safety control method, device, equipment and medium. The method comprises the following steps: according to a triggered target task, an operation sequence, a minimum permission token and an expected page state sequence generated through a pre-preparation stage are acquired; each action in the operation sequence is executed in sequence, including judging whether a target object of the action is within an operation object range defined by the minimum permission token, and initiating secondary confirmation if user confirmation is required; if confirmation is not required or has been obtained, a difference between a current real UI state and a corresponding expected page state is checked; if the difference checking fails, the execution is suspended and re-preparation is triggered; if the difference checking is passed, the action is executed, and a corresponding audit event is recorded to generate a replayable audit chain. Through the full-link safety control of token authorization, user confirmation, state checking and audit solidification, the application solves the problems of misoperation and unauthorized execution in intelligent agent automatic operation.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence technology, specifically to a method, device, equipment, and medium for safe operation control of a terminal intelligent agent. Background Technology

[0002] A common security challenge when terminal intelligent agents operate applications on behalf of users is that automated execution may trigger irreversible actions such as payments, deletions, and sending, and operation permissions are difficult to pinpoint to specific interface elements. In existing technologies, intelligent agents typically generate and execute clicks, inputs, etc., directly based on the recognition results of the current interface, with system-granted operation permissions at the application-wide granularity. When the interface changes (e.g., button text changes, amount changes, pop-ups appear), the intelligent agent continues to execute according to the original operation sequence. Furthermore, existing systems only record the action name and timestamp, failing to fully preserve the interface state at the time of the operation and the intelligent agent's judgment criteria. The consequences of this approach are: the intelligent agent may operate on controls unrelated to the current task, continue executing the original action even after key interface information changes, and it is difficult to reconstruct the operation process to analyze the cause after a misoperation, increasing the risk of unknown behavior in intelligent agent-authorized operations during terminal operation.

[0003] Therefore, how to improve the manageability of terminal intelligent agents' behavior during automated operation and further reduce the operational risks of terminal intelligent agents in automated operation is an urgent technical problem to be solved. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention provide a method, apparatus, device and medium for safe control of terminal intelligent agent operation, in order to solve the problem of how to improve the manageability of behavior of terminal intelligent agents in the process of automatic operation and further reduce the operational risks of terminal intelligent agents in automated operation.

[0005] According to one aspect of the present invention, a terminal intelligent agent operation security control method is provided, comprising: Based on the target task triggered by the user, obtain the operation sequence, the least privilege token, and the expected page state sequence generated through the pre-rehearsal stage; wherein, the operation sequence includes at least one action, and the least privilege token at least limits the scope of operations allowed to be performed in the target task, the validity period, and the types of actions that require user confirmation; Each action in the operation sequence is executed sequentially, including determining whether the target object of the action is within the scope of the operation object limited by the least privilege token; wherein, if the target object of the action is within the scope of the operation object and the type of the action is an action type that requires user confirmation, then the execution is paused and a secondary confirmation operation is initiated to the user; if the type of the action is not an action type that requires user confirmation or has already obtained secondary confirmation from the user, then the difference between the current real UI state and the corresponding expected page state is verified. If the difference verification fails, execution is paused and a re-render is triggered; if the difference verification passes, the action is executed, the corresponding audit event is recorded, and a replayable audit chain is generated.

[0006] Furthermore, the pre-rehearsal stage includes: Obtain the interface information of the current foreground application on the mobile terminal, identify interactive control nodes from the interface information, and obtain a set of control nodes; Based on the control node set, a UI semantic graph is constructed, and the nodes in the UI semantic graph have business semantics and operation attribute information; Based on the target task triggered by the user and the UI semantic graph, a candidate operation sequence based on the target task is generated; The candidate operation sequence is input into the sandbox pre-simulation state machine, and each action contained in the candidate operation sequence is processed in sequence. For actions whose operation attribute information is a set of security types, the expected page state after execution is collected. For actions whose operation attribute information is a set of risk types, no real business action is executed, and they are determined to be action types that require user confirmation. The expected page states collected through the pre-rehearsal and the action types identified as requiring user confirmation are merged into the pre-rehearsal result; based on the pre-rehearsal result, the operation sequence, the least privilege token, and the expected page state sequence uniquely bound to the target task are generated.

[0007] Further, the step of obtaining the interface information of the current foreground application of the mobile terminal, identifying interactive control nodes from the interface information, and obtaining a set of control nodes includes: Obtain the interface information, which includes a control tree, screenshots, and accessibility node information; Interactive control nodes are extracted from the control tree and the accessibility node information to generate an initial set of control nodes; Visual recognition is performed on the screenshot, including OCR text recognition and icon semantic recognition; The results of the visual recognition are fused with the initial set of control nodes. For nodes that already exist in the initial set of control nodes, the recognition results are used to supplement or correct the corresponding operation attribute information. For nodes that are recognized but do not exist in the initial set of control nodes, new control nodes are created according to the corresponding node position and interaction type. The set of control nodes obtained through fusion will be used as the enhanced set of control nodes.

[0008] Furthermore, the security type set includes actions without external side effects or reversible changes; the risk type set includes actions that generate external network requests, fund changes, or permission changes; the sandbox pre-simulation state machine adopts at least one pre-simulation mode among virtual pre-simulation, shadow page pre-simulation, and controlled real-world detection pre-simulation; wherein, the virtual pre-simulation simulates page state changes based on UI semantic graphs and page state transition models; the shadow page pre-simulation executes candidate operation sequences in an isolated environment; the controlled real-world detection pre-simulation executes only actions belonging to the security type set in a preset restricted real page to detect subsequent page states.

[0009] Furthermore, the audit events corresponding to the records generate a replayable audit chain, including: The audit event is generated by recording the currently completed action. Obtain the hash value of the previous audit event, write it into the audit event, and perform a hash calculation on the audit event to which the hash value of the previous audit event was written, to obtain the hash value of the audit event. The audit events with the hash values ​​are linked sequentially according to the execution order of the actions. Once the audit events corresponding to all actions are linked, the replayable audit chain is obtained.

[0010] Furthermore, after pausing execution and triggering a re-render, the process also includes: Obtain the execution status information at the time of pause and generate a manual takeover interface, wherein the manual takeover interface includes prompts for undone operations and corresponding undone entry points; In response to a user-triggered undo operation, execute the corresponding rollback process; or in response to a user-selected re-preview instruction, re-execute the pre-preview stage based on the current real UI state. If the preview passes, output the updated operation sequence, the least privilege token, and the expected page state sequence, and continue execution from the point of interruption.

[0011] Furthermore, the verification of the difference between the current actual UI state and the corresponding expected page state includes: Obtain information on the differences between the current actual UI state and the corresponding expected page state across multiple preset dimensions; Calculate the overall difference value based on the weights corresponding to each preset dimension; If the overall difference value does not exceed the preset threshold and no difference appears in the preset key dimensions, the difference verification is deemed to have passed; if any key preset dimension shows a difference, the difference verification is deemed to have failed.

[0012] According to another aspect of the present invention, a terminal intelligent agent operation security control device is provided, comprising: The data acquisition module is used to acquire the operation sequence, the least privilege token, and the expected page state sequence generated through the pre-rehearsal stage according to the target task triggered by the user; wherein, the operation sequence includes at least one action, and the least privilege token at least limits the scope of operation objects allowed to be executed in the current target task, the validity period, and the type of action that requires user confirmation; The execution verification module is used to execute each action in the operation sequence in sequence, including determining whether the target object of the action is within the scope of the operation object limited by the least privilege token; wherein, if the target object of the action is within the scope of the operation object and the type of the action is an action type that requires user confirmation, then the execution is paused and a secondary confirmation operation is initiated to the user; if the type of the action is not an action type that requires user confirmation or has already obtained secondary confirmation from the user, then the difference between the current real UI state and the corresponding expected page state is verified. The result generation module is used to pause execution and trigger a re-run if the difference verification fails; if the difference verification passes, the action is executed and the corresponding audit event is recorded to generate a replayable audit chain.

[0013] According to another aspect of the present invention, a computer device is provided, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; The memory is used to store at least one executable instruction, which causes the processor to perform the operation of the terminal intelligent agent operation security control method described in any of the above-described methods.

[0014] According to another aspect of the present invention, a computer-readable storage medium is provided, the storage medium storing at least one executable instruction, the executable instruction causing a computer device to perform the operation of the terminal intelligent agent operation security control method described in any one of the preceding embodiments.

[0015] According to another aspect of the present invention, a computer program product is provided, including a computer program that, when executed by a processor, implements the terminal intelligent agent operation security control method described in any of the above embodiments.

[0016] This invention, through pre-execution, acquires the operation sequence, minimum authority token, and expected page state sequence generated during the pre-rehearsal stage, forcibly binding the agent's execution behavior with the pre-rehearsal verification results. The minimum authority token limits the scope of the operation object, its validity period, and the type of action requiring confirmation on a task-by-task basis, ensuring the agent operates only within the token's authorized scope and preventing unauthorized execution. At each step, the target object is checked to ensure it falls within the token's scope; if not, execution stops immediately to prevent out-of-bounds operations caused by page drift. For action types within the token's scope but marked as requiring confirmation, a forced pause and a secondary confirmation interface are displayed, ensuring high-risk actions require conscious user authorization. For confirmed or non-confirmation-required actions, the difference between the current actual UI state and the expected state is further verified, and a re-prehearsal is triggered based on the difference, enabling the system to adaptively adapt to dynamic interface changes. After each step, an audit event is recorded and a hash chain-like audit chain is generated; any tampering with any event results in subsequent hash matching failures, supporting precise post-event replay and accountability. By constructing a closed-loop security control system across four stages—token authorization, user confirmation, status verification, and audit consolidation—the system resolves issues of misoperation and unauthorized operations caused by model illusion, interface misidentification, excessive permission granting, or page drift.

[0017] The above description is merely an overview of the technical solutions of the embodiments of the present invention. In order to better understand the technical means of the embodiments of the present invention and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0018] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating the terminal intelligent agent operation security control method provided in an embodiment of the present invention is shown. Figure 2 This diagram illustrates the structure of a terminal intelligent agent operation safety control device provided in an embodiment of the present invention. Figure 3 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention is shown. Detailed Implementation

[0019] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0020] Figure 1A flowchart of the terminal intelligent agent operation security control method provided in an embodiment of the present invention is shown, such as... Figure 1 As shown, the method includes the following steps: S110. Based on the target task triggered by the user, obtain the operation sequence, the least privilege token, and the expected page state sequence generated through the pre-rehearsal stage; wherein, the operation sequence includes at least one action, and the least privilege token at least limits the scope of operations allowed to be executed in the target task, the validity period, and the types of actions that require user confirmation.

[0021] When a user triggers a target task (e.g., "Send this invoice to Xiao Li in finance") through a terminal device (such as a smartphone or tablet), the system does not immediately instruct the agent to perform the operation. Instead, it first obtains the execution plan and security credentials generated after a security analysis of the task in a "pre-operation rehearsal phase" before the actual operation. This binds the agent's execution behavior to the security assessment results of the rehearsal phase, preventing the agent from deviating from the security boundaries of the rehearsal verification during actual execution, thus providing a unified decision-making basis for subsequent permission and status verification.

[0022] Specifically, the "pre-implementation simulation phase" involves simulating and assessing the risks of a series of operations planned by the agent within a sandbox state machine isolated from the real business environment. Upon completion of this phase, three key data points will be output: Firstly, the operation sequence. The operation sequence is an ordered list of actions planned by the agent to complete the target task. It contains multiple actions arranged in execution order. Each action can be a basic operation unit such as clicking, inputting, swiping, selecting, calling a tool, or waiting. In this embodiment, a single execution step in the operation sequence is referred to as an "action." By structurally decomposing the target task in the form of an operation sequence, the user's natural language target can be transformed into a sequence of instructions that can be executed by the terminal device. This allows the agent to proceed in an orderly manner according to the pre-verified path, avoiding execution chaos caused by misunderstandings of the target or planning errors.

[0023] Secondly, the token of least privilege. This token of least privilege is the core security credential during the execution of this task, defining all operational boundaries of the intelligent agent when performing this target task. Specifically, the token of least privilege at least defines: The scope of allowed operations: This scope precisely defines the controls that the agent can interact with, specifically a set consisting of application identifiers, page signatures, control resource identifiers, or stable control semantic fingerprints. During execution, the agent can only operate on controls such as buttons, input boxes, or list items belonging to this set. In one implementation, if the target control of the currently executed action cannot be precisely matched within the scope of operations, the system enters a candidate control rebinding process: Candidate controls must be located in the same application, the same or equivalent page, and the same business object area, and their roles, action types, business semantics, and risk levels must not increase; the system calculates a matching score based on text similarity, visual location similarity, hierarchical path similarity, adjacent label similarity, business object consistency, and historical binding success rate; if the matching score is higher than the safety threshold and the risk level remains unchanged, the control is rebound and execution continues; if only moderate similarity is found, or if it involves key business objects such as amounts, contacts, orders, files, or authorizations, or if the risk level increases, execution is paused and a re-render or user confirmation is requested; if no match can be found or multiple similar candidates appear, execution is directly rejected. Therefore, even in extreme cases (such as when the agent's underlying planning model generates unexpected operational intentions, or when the underlying accessibility capabilities are improperly invoked, or when application updates cause changes in control identifiers), the least privilege token can still serve as a last line of defense, strictly constraining the agent's actual behavioral capabilities to the set of controls that have passed pre-rehearsal verification or equivalent controls that have been re-bound and verified, thereby preventing the agent from "overstepping its authority" to operate any interface elements that have not been pre-rehearsed and verified during the actual execution phase, and avoiding the risk of accidental clicks caused by model illusion, interface misidentification, or page drift.

[0024] Validity Period: The least privilege token has an expiration time, containing an expiration timestamp. Once this validity period expires, the least privilege token will automatically become invalid, and all operations of the agent will be prohibited. This prevents the token from being used in unexpected contexts due to user abandonment, long task suspension, or interference from other applications, and avoids abuse of the token due to prolonged validity or its use for operations unrelated to the current target task.

[0025] Actions requiring user confirmation: The least privilege token explicitly specifies which types of actions (e.g., "send message," "submit order," "delete data," "payment confirmation") require secondary user confirmation before execution. Therefore, the least privilege token not only limits "what can be done," but also "which actions cannot be performed automatically and must be confirmed by a human," ensuring that high-risk business actions require conscious user authorization before execution, preventing the intelligent agent from making decisions with significant business consequences on behalf of the user without their knowledge.

[0026] Third, the expected page state sequence. This expected page state sequence corresponds to the operation sequence and records the expected mobile interface state after each simulated action in the pre-rehearsal phase. Examples include expected page icons, key controls and their text content, and visible business objects (such as amounts and contacts). This provides a pre-calculated "expected anchor point" for each action in the actual execution phase, ensuring a reference benchmark for subsequent state difference verification. This solves the problem in existing technologies where the lack of an execution-period reference system makes it impossible to determine whether the execution deviates from expectations.

[0027] Through this step, the system anchors the agent's behavioral capabilities within a safe and controllable framework from the pre-rehearsal stage, realizing a paradigm shift from "agent's free planning and execution" to "controlled execution under pre-rehearsal constraints," significantly reducing the safety risks of the agent's automatic operation.

[0028] S120. Execute each action in the operation sequence in sequence, including determining whether the target object of the action is within the scope of the operation object limited by the least privilege token; wherein, if the target object of the action is within the scope of the operation object and the type of the action is an action type that requires user confirmation, then pause execution and initiate a secondary confirmation operation to the user; if the type of the action is not an action type that requires user confirmation or has already obtained secondary confirmation from the user, then verify the difference between the current real UI state and the corresponding expected page state.

[0029] After acquiring the data output from the pre-rehearsal stage, the system initiates the actual execution process and begins to process each action in the operation sequence sequentially.

[0030] First, before executing each action, the system performs a strict operation object scope verification. Specifically, the system reads the target object of the action to be executed and compares it with the scope of allowed operations defined in the minimum permission token. Only if the target object exists within the scope of operations is the system considered qualified to execute the current action; otherwise, execution is immediately stopped and an exception handling process is triggered. In one implementation, if the target control cannot be precisely matched but there are approximate candidate controls that meet the above conditions (same application, same equivalent page, same business object area, role and risk level not increased), the system can rebind it according to the above matching score calculation method and record the reason for rebinding and the matching score in the audit event; if the matching score does not reach the security threshold, execution is rejected. Thus, the legality of each action can be verified in real time before execution, preventing the agent from attempting to operate interface elements that have not been authorized in the pre-rehearsal due to dynamic changes in the interface (such as page drift after the pre-rehearsal or changes in control identifiers), thereby minimizing the risk of unauthorized access during the execution phase.

[0031] After determining that the target object of the action belongs to the scope of the operation object limited by the least privilege token, the system will further determine whether the type of the action belongs to the type of action that requires user confirmation, that is, check whether the type of the action matches the "type of action that requires user confirmation" in the least privilege token.

[0032] If the current action type matches the type requiring user confirmation, the system will trigger an interaction control mechanism. Specifically, the system will pause the agent's automatic execution process and display a secondary confirmation interface to the user's terminal. This secondary confirmation interface must be designed to be unclickable or unskippable by the agent, and its content should clearly display details of the upcoming action, including but not limited to the action name, the target audience, the expected business consequences, and whether it is reversible, allowing the user to make an informed decision. The system waits for the user to make a clear "confirm" or "decline" selection on this interface. This forces the user to actively judge before high-risk actions such as payment, sending, deletion, authorization, and submission are executed, ensuring that any operation with business consequences such as changes in funds, external dissemination, or irreversible data alteration is consciously authorized by the user. This avoids the security vulnerability of existing automation solutions that only solicit authorization once at the start of the task, leading to a lack of item-by-item supervision for subsequent high-risk operations. Furthermore, displaying business consequences rather than abstract permission descriptions on the confirmation interface reduces the user's cognitive burden and improves the accuracy and efficiency of confirmation decisions.

[0033] If the current action does not belong to the type of action requiring user confirmation, or if the action, although belonging to the type of action requiring user confirmation, has already received secondary confirmation from the user, then the system enters the final security check before execution, namely, performing a state difference check. Therefore, completing the user confirmation check before the state difference check ensures that high-risk actions are confirmed before entering the environment adaptation check, avoiding unnecessary computational resource consumption for high-risk actions without user authorization. It also decouples the triggering conditions of the state difference check from the user confirmation strategy, improving the flexibility and execution efficiency of the check process.

[0034] The execution state difference verification specifically involves the system collecting the current "real UI state" through the mobile phone environment perception module and comparing it with the corresponding expected page state in the "expected page state sequence" obtained from the pre-rehearsal stage. Both the real UI state and the expected page state include multi-dimensional information such as page structure, control text, control layout, business objects (e.g., monetary value, contact name), and risk tags. This allows for quantitative verification of the consistency between the virtual environment or controlled detection environment during pre-rehearsal and the current real execution environment before each action is actually executed. This aligns the "safety expectation" established during the pre-rehearsal stage with the "actual state" of the real world, effectively solving the impact of page drift caused by dynamic factors such as App version updates, A / B testing, personalized recommendations, and pop-up ads on the correctness of the agent's execution.

[0035] S130. If the difference verification fails, the execution is paused and a re-render is triggered; if the difference verification passes, the action is executed and the corresponding audit event is recorded to generate a replayable audit chain.

[0036] After the state difference verification is completed before execution, the system performs branch processing based on the verification results.

[0037] On the one hand, if the difference verification fails, it indicates an unacceptable deviation between the current real mobile interface and the expected interface during the pre-rehearsal (for example, the button text changes from "Save Draft" to "Pay Now," or the payment amount changes from "100 Yuan" to "1000 Yuan"). In this case, the system determines that the current execution environment has deviated from the scope of the security pre-rehearsal. To prevent accidental operation, the system will immediately pause the execution process and trigger a re-prehearsal process. The re-prehearsal process uses the current "real UI state" as the new starting state and re-executes the analysis from the pre-prehearsal stage to generate an operation sequence adapted to the new interface, a new least privilege token, and a new expected page state sequence. This allows the system to adaptively re-evaluate security when faced with interface changes, rather than simply terminating or blindly continuing, to adapt to dynamic interface changes with minimal execution cost. This minimizes the risk of execution errors caused by interface drift and avoids the inefficiency of traditional solutions where users must completely restart the task once an error occurs.

[0038] On the other hand, if the difference verification passes, it indicates that the current real interface state is consistent with the pre-simulation expectation, and the risk is controllable. At this time, the system will authorize the execution of the action. The agent calls the terminal system's accessibility service or simulated input interface to complete operations such as clicking, inputting, or swiping on the real mobile phone interface. In this way, executing the action under the premise of passing the difference verification ensures that every step of the agent's operation is based on the dual security guarantee of "pre-simulation verification and environment confirmation," thereby maintaining the efficiency of the agent's automated execution while reducing the probability of misoperation to an acceptable safety level.

[0039] After executing the action, the system immediately records the execution process to generate traceable audit evidence. Specifically, the system generates an audit event corresponding to the current action. The audit event is a structured data object, which includes at least the timestamp of the executed action, action type, target object identifier, action execution result, and a summary of the current actual UI state. In one implementation, the audit event also includes information such as task identifier, operation sequence identifier, token identifier, expected page state hash value, actual page state hash value, difference score, risk level, the reason the model selected the action, token verification result, and user secondary confirmation record, so as to fully reconstruct the execution context in subsequent audit playback. Therefore, execution evidence can be promptly solidified after the action is executed, providing the original data foundation for subsequent audit playback and anomaly tracing, and avoiding the inability to reconstruct the execution scene due to lost execution state.

[0040] Subsequently, the system links the newly generated audit events to the existing audit chain, creating a replayable audit chain. This replayable audit chain strings together various audit events according to the execution time sequence of the actions, forming a complete and tamper-proof record of evidence. This replayable audit chain not only records "what happened," but more importantly, it records key security contexts such as "under what interface state" and "whether secondary user confirmation was obtained," thus supporting accurate replay and compliance auditing of all actions performed by the agent afterward. This allows users to fully review the entire execution process after the agent completes its work—including the interface state at each step, the agent's decision-making rationale, the verification status of the permission token, user confirmation nodes, and the execution result—solving the problem in existing technologies where insufficient operation log information prevents the reconstruction of the full execution picture, making accountability and review difficult. This provides reliable technical support for enterprise compliance auditing, user trust building, and continuous system optimization.

[0041] In one embodiment, the pre-rehearsal stage includes: S100: Obtain the interface information of the current foreground application of the mobile terminal, identify interactive control nodes from the interface information, and obtain a set of control nodes; S101. Based on the control node set, construct a UI semantic graph, wherein the nodes in the UI semantic graph have business semantics and operation attribute information; S102. Generate a candidate operation sequence based on the target task triggered by the user and the UI semantic graph; S103. Input the candidate operation sequence into the sandbox pre-simulation state machine, and process each action contained in the candidate operation sequence in sequence. For actions whose operation attribute information is a set of security types, execute and collect the expected page state after execution. For actions whose operation attribute information is a set of risk types, do not execute the actual business action and determine it as an action type that requires user confirmation. S104. The expected page states collected through the pre-rehearsal and the action types identified as requiring user confirmation are merged into a pre-rehearsal result; based on the pre-rehearsal result, the operation sequence, the least privilege token, and the expected page state sequence uniquely bound to the target task are generated.

[0042] In this embodiment, before the actual execution phase begins, the system first performs a pre-rehearsal phase to conduct security analysis and risk verification of the operations planned to be executed by the agent.

[0043] Specifically, the system activates the mobile environment awareness module to obtain the interface information of the current foreground application on the mobile terminal. This interface information includes a control tree, screenshots, and accessibility node information. The system extracts interactive control nodes—including clickable buttons, input boxes, selectable checkboxes and radio buttons, scrollable list containers, etc.—from the control tree and accessibility node information to obtain a control node set. Each control node records at least a resource identifier, control type, displayed text, content description, boundary position, and accessibility role. In one implementation, the system can also perform visual recognition (including OCR text recognition and icon semantic recognition) on the screenshots and fuse the visual recognition results with the initial control node set extracted from the control tree. The recognition results are used to supplement or correct the operational attribute information of the corresponding nodes, or new control nodes are created based on the identified interactive areas, resulting in an enhanced control node set to compensate for the semantic information deficiencies of the pure control tree.

[0044] Based on the aforementioned set of control nodes, the system constructs a UI semantic graph. The UI semantic graph includes nodes and edges: node types include page nodes, control nodes, field nodes, and pop-up nodes; edge types include executable action edges, page navigation edges, field dependency edges, and risk edges. Each operable node has business semantics and operational attribute information. The business semantics characterize the functional meaning of the node from a business perspective, such as "submit order," "unsubscribe," "send message," "delete file," "next step," and "return to homepage." The identification of business semantics adopts a multi-source fusion comprehensive judgment mechanism: keyword matching serves as a low-cost first-layer recall mechanism; based on this, it integrates multi-dimensional evidence such as control tree attributes (control type, resource identifier, content description), OCR-recognized text, icon category (inferred through icon recognition model), color and visual emphasis features (e.g., red warning button, emphasized main button), control position (whether it is at the bottom main operation position, whether it is close to key areas such as amount / recipient / authorization instructions), page title (e.g., "order confirmation page", "payment page", "authorization application page"), adjacent label text, business object fields (e.g., amount field, contact person field, product specification field), AppIntent / shortcut / accessibility node capability description, historical page transition relationship (which page jumps to after the control is clicked), observable network request features (whether it points to interfaces such as create / submit / pay / send / delete / authorize), user goal constraints, and model semantic understanding results to generate business semantics and its confidence score. For example, for a button that only displays "OK", the keyword itself cannot determine the risk. The system will consider the context, such as whether the page it is on is an order confirmation page, whether the adjacent text contains amount / recipient / product / authorization instructions, whether the button is in the bottom main operation position, whether the preceding page is a payment process, and whether clicking it may call the payment SDK or submit API, to comprehensively determine whether it is payment confirmation, order submission, permission granting, or ordinary confirmation. Operation attribute information should at least include role information (e.g., button, input box, switch), side effect type (used to characterize the business impact that executing the operation may have, such as no side effects, local modification, network submission, external sending, fund action, permission change), risk level, reversibility, and whether user confirmation is required.The reversibility determination is derived from a fusion of three layers: the first layer is static prior, which is based on control roles, text, icons, page titles, adjacent prompts, AppIntentSchema, system permission pop-up types, historical page transition relationships, and pre-defined business rules for initial determination; the second layer is runtime context, which dynamically corrects the determination by combining the current App identifier, page identifier, login account type, business object status (e.g., summaries of objects such as amount, contact, order, and file), whether it is on a draft page or confirmation page, and whether there are compensable entry points such as "withdraw / cancel / restore / save draft"; the third layer is execution feedback, which is based on subsequent page information collected after sandbox pre-running or controlled probing, network or system prompts, the existence of the withdrawal entry point, and its validity period for final verification. For specific applications and specific pages, the system maintains a reversibility configuration file, recording the application identifier, page signature, control identifier, action type, business object type, observed withdrawal entry point, withdrawal time window, compensatory action, confidence score, and final verification time to support dynamic updates of the reversibility determination. The aforementioned business semantics and operational attribute information are based on the control tree, supplemented by visual recognition, and based on the page context for disambiguation. After multi-source fusion, each control node is assigned a corresponding attribute value.

[0045] After constructing the UI semantic graph, the system generates candidate operation sequences based on the user-triggered target task and the UI semantic graph. Specifically, the system first transforms the user-triggered target task into a structured target, which includes at least a task identifier, the user's original target text, a set of allowed applications, the target object, completion conditions, prohibited actions, and user confirmation policies. Subsequently, the system uses the current page node as the starting node and the completion condition as the target state, searching the UI semantic graph for one or more paths from the current state to the target state. Each step in the path is an action, and these actions are arranged sequentially to form a candidate operation sequence. During the search, operation sequences containing prohibited actions are filtered, and sequences containing high-risk nodes are marked and retained.

[0046] After generating candidate operation sequences, the system inputs these sequences into a sandbox pre-operation state machine for security pre-operation. The sandbox pre-operation state machine processes each action sequentially, first reading the operation attribute information of the target control for the current action to determine whether it belongs to the safe type set or the risk type set. The safe type set includes actions without external side effects or reversible changes, specifically including: observation actions (only reading interface information without changing any state), locally reversible actions (only modifying local UI state or local temporary data and can be undone), and process advancement actions (mainly used for page navigation or form pagination, which do not directly result in external submissions, fund changes, permission grants, or irreversible data modifications). The risk type set includes actions that generate external network requests, fund changes, or permission changes, specifically including: high-risk business actions (e.g., submitting forms, sending messages, deleting data), strong-risk actions (e.g., payments, transfers, authorizing new permissions, public releases), and uncertain actions (semantically ambiguous "confirm," "complete," "agree," etc., whose business consequences the system cannot determine). It should be noted that the distinction between process advancement actions in the safe type set and high-risk business actions in the risk type set is not solely based on button names. In one implementation, the system employs a dynamic risk assessment mechanism: when it detects that the page has collected complete business objects (such as amount, order, recipient, address, approval opinion), the text near the button contains keywords such as "submit," "confirm," "save," "place order," "create," "send," "publish," "delete," "authorize," and "pay," there is dirty form data before clicking and the application history shows that "next" will be automatically saved, the button has caused the generation of order number / work order number / message ID / approval serial number during historical page transitions, or observable network requests point to interfaces such as create / submit / pay / send / delete / authorize, even if the button is displayed as "next" or "continue," the system will dynamically upgrade the action from the safe type set to the risk type set, prohibiting its automatic execution in controlled real-world probing, thereby avoiding risk misjudgment caused by ambiguous interface text.

[0047] For actions belonging to the safety type set, the system executes the action in the sandbox pre-simulation state machine and collects the expected page state after execution. For actions belonging to the risk type set, the system does not execute the actual business action, but instead identifies it as an action type requiring user confirmation and writes it into the side effect ledger and risk ledger. The sandbox pre-simulation state machine can adopt at least one mode among virtual pre-simulation, shadow page pre-simulation, or controlled real-world probing pre-simulation. In one implementation, the selection strategy for the three modes is as follows: if a page flow model or structured tool description exists, virtual pre-simulation is preferred; if the target application provides a sandbox page, a copy of test data, a draft container, or an enterprise-controlled environment, shadow page pre-simulation is used; if none of the above conditions are met, controlled real-world probing pre-simulation is used. When using controlled real-world probing rehearsals, for third-party applications such as banks, social media, and e-commerce, the system defaults to not using real-world probing as a free exploration method. Instead, it first conducts virtual rehearsals based on historical page models, the current UI semantic graph, public page structures, user local historical paths, rule bases, and known application processes. Only when the information on the next page is necessary to complete the task, and the action to be performed is assessed as low-risk, reversible, or without external side effects, is a single or limited number of real-world page advances permitted using a probe token. The system performs an AntiAutomationRisk assessment, with assessment factors including application type and sensitivity level, whether the account is a real personal account, whether the page involves login / payment / finance / social dissemination, whether CAPTCHA / slider / face / SMS verification / risk control prompts appear, operation frequency per unit time, depth of continuous page jumps, whether it requires reading or inputting sensitive credentials, and whether it triggers system security pop-ups. If the assessment exceeds a preset threshold, real-world probing will cease, and the system will switch to virtual rehearsals or request the user to manually advance. The controlled real-world detection simulation also sets hard boundaries: maximum number of detection steps, maximum page depth, maximum dwell time, minimum click interval, prohibition of bypassing CAPTCHA, prohibition of simulating biometrics / payment passwords / SMS verification codes, prohibition of high-risk actions such as payment submission / sending / deletion / authorization / public posting, and if risk control prompts, abnormal login reminders, frequent operation prompts, or new permission requests appear during the detection process, it will be stopped immediately and manual takeover will be initiated, and the actions, reasons, page summaries, and reasons for stopping each detection will be recorded in the audit chain.

[0048] The boundaries of the sandbox rehearsal are: allowing low-risk, reversible, or non-external side-effect-free actions to proceed in a controlled manner to discover subsequent page states and confirmation nodes; while requiring the truncation, registration, and confirmation of actions that will have external business consequences (such as network submissions, fund changes, permission changes, external dissemination, etc.), without executing real business logic, thereby achieving a balance between security and information integrity in the rehearsal phase.

[0049] After the sandbox pre-simulation state machine completes all action processing, the system generates pre-simulation results. These results include at least the actions executed under control and their corresponding expected page states, the types of actions requiring user confirmation and their potential side effects, a list of risk nodes, a set of uncertainties, and a recommended execution strategy. The system summarizes the expected page states collected during the pre-simulation into an expected page state sequence and summarizes the types of actions requiring user confirmation into a confirmation requirement list. Subsequently, based on the pre-simulation results, the system generates three outputs uniquely bound to the target task: an operation sequence (the actual sequence of actions passed during the pre-simulation); an expected page state sequence, corresponding one-to-one with each action in the operation sequence, serving as a reference for verifying differences in the actual execution phase; and a minimum permission token, generated based on the pre-simulation results, which includes at least the scope of allowed operations, the scope of allowed fields to be written, the scope of allowed tools to be invoked, the types of actions requiring user confirmation, the validity period, the maximum number of execution steps, and business-level constraints, and is digitally signed by the local security module. The operation sequence, the least privilege token, and the expected page state sequence together constitute the secure execution foundation for the actual execution phase, ensuring that the agent remains in a controlled state that has passed the pre-rehearsal verification during subsequent execution.

[0050] In one embodiment, obtaining the interface information of the current foreground application of the mobile terminal, identifying interactive control nodes from the interface information, and obtaining a set of control nodes includes: S1110. Obtain the interface information, which includes a control tree, screenshot, and accessibility node information. S1120. Extract interactive control nodes from the control tree and the accessibility node information to generate an initial control node set; S1130. Perform visual recognition on the screenshot, the visual recognition including OCR text recognition and icon semantic recognition; S1140. The result of the visual recognition is fused with the initial control node set, wherein, for nodes that already exist in the initial control node set, the corresponding operation attribute information is supplemented or corrected using the recognition result; for nodes that are recognized but do not exist in the initial control node set, new control nodes are created according to the corresponding node position and interaction type. S1150, Use the control node set obtained through fusion as the enhanced control node set.

[0051] In this embodiment, the system first acquires the interface information of the current foreground application on the mobile terminal. This interface information includes a control tree provided by the system's accessibility framework, a screenshot of the current screen, and accessibility node information. The control tree contains the hierarchical structure, control types, text content, and boundary coordinates of all visible view elements in the current interface. The accessibility node information provides indicators of interactive capabilities such as whether each node is clickable, input-enabled, focusable, and scrollable. The screenshot is used for subsequent visual supplementary recognition. The system extracts interactive control nodes from the control tree and the accessibility node information—including clickable buttons, input boxes, selectable checkboxes and radio buttons, scrollable list containers, etc.—to generate an initial control node set. Each initial control node records at least its resource ID, control type (ClassName), display text (Text), content description (ContentDescription), boundary position (Bounds), and accessibility role (Role). Simultaneously, the system performs visual recognition on the screenshot, including OCR text recognition and icon semantic recognition. Specifically, OCR text recognition identifies all visible text areas in a screenshot, along with their corresponding text content and coordinates. It can capture interface areas in the control tree that may be missing or empty but visually contain text labels. Icon semantic recognition uses a pre-trained icon classification model to categorize non-text areas in the screenshot, identifying icon elements in the interface and inferring their semantic categories, such as "back arrow," "search magnifying glass," "more menu," "close cross," "add," "favorite," and "share." After obtaining the visual recognition results, the system fuses these results with the initial set of control nodes. The fusion process matches based on the overlap of boundary positions: when the overlap between a recognition area in the visual recognition result and the boundary position of a node in the initial control node set exceeds a preset threshold (e.g., the overlap area exceeds 50%), the system determines that the two correspond to the same interface element. At this time, for the existing node, the system uses the recognition result to supplement or correct the corresponding operation attribute information. For example, when a control node in the control tree only contains a resource identifier without display text, the OCR recognition result can be used to supplement its text content; when a pure icon button in the control tree lacks semantic description, the icon semantic recognition result can be used to supplement its semantic category information; when the display text of a control is truncated or unreadable in the control tree, the OCR recognition result can be used to correct its complete text.When a certain recognition region in the visual recognition result does not have a matching node in the initial control node set, the system determines that the corresponding interface element is not exposed by the control tree or accessibility node information. In this case, a new control node is created based on the boundary position and interaction type of the region (e.g., the OCR recognition region is located in a visually clickable image or icon area, or its interactivity is inferred through a deep learning model). The system then assigns the text or semantic information obtained from visual recognition as its operational attribute information. Finally, the control node set obtained through fusion is used as the enhanced control node set. This enhanced control node set retains the structured precision, hierarchical relationships, and system-level interactive capability identifiers of the control tree, while supplementing it with the text content and icon semantic information obtained from visual recognition. This allows interface elements that were previously unrecognizable due to missing control tree information (e.g., custom controls not exposing semantics) or visual interactive elements not present in the control tree to be included in subsequent processing, providing a more complete data foundation for the accurate annotation of business semantics and operational attribute information in UI semantic graph construction.

[0052] In one embodiment, the security type set includes actions without external side effects or reversible changes; the risk type set includes actions that generate external network requests, fund changes, or permission changes; the sandbox pre-simulation state machine employs at least one pre-simulation mode among virtual pre-simulation, shadow page pre-simulation, and controlled real-world detection pre-simulation; wherein, the virtual pre-simulation simulates page state changes based on UI semantic graphs and page state transition models; the shadow page pre-simulation executes candidate operation sequences in an isolated environment; and the controlled real-world detection pre-simulation executes only actions belonging to the security type set in a preset restricted real page to detect subsequent page states.

[0053] In this embodiment, when the sandbox pre-simulation state machine processes each action in the candidate operation sequence, it first determines whether the action belongs to the safe type set or the risk type set based on the operation attribute information of the target control of the action (especially the side effect type and risk level). The safe type set includes actions without external side effects or reversible changes, that is, the execution of the action will not have a lasting impact on the external business system, or even if changes occur, they can be restored by undoing the operation. Specifically, in one implementation, the safe type set can be subdivided according to risk level into: observation actions (E0), such as observing a page, scrolling a list, expanding a collapsible panel, this type of action only reads interface information without changing any state; local reversible actions (E1), such as filling in form fields, selecting drop-down options, checking checkboxes, this type of action only modifies the local UI state or local temporary data, and can be undone by clearing, deselecting, or returning; process advancement actions (E2), such as clicking "Next," "Continue," "Preview," "Save Draft," this type of action is mainly used for page navigation or form pagination advancement, and it does not directly result in external submission, fund changes, permission granting, or irreversible data modification. The risk type set includes actions that generate external network requests, fund transfers, or permission changes. Specifically, the execution of such actions will lead to changes in server-side data status, involve fund transfers or payments, or trigger the granting or modification of system permissions. In one implementation, the risk type set can be further subdivided according to risk level: High-risk business actions (E3), such as submitting forms, sending messages, canceling orders, deleting data, and external sharing. These actions generate external network requests and server-side status changes, and are usually irreversible or can only be partially withdrawn within a limited time window; Strong-risk actions (E4), such as payments, transfers, authorizing new permissions, public publishing, irreversible deletion, and activating automatic renewal. These actions involve funds, permissions, or public dissemination, with serious business consequences that are usually irreversible; and Uncertain actions (EU), such as semantically ambiguous "confirm," "complete," "agree," and "continue," whose specific business consequences the system cannot determine from the current context.

[0054] It should be noted that the distinction between E2 and E3 is not solely based on the button name. In one implementation, the system employs a dynamic risk assessment mechanism: before actual execution, the system reads multiple signals for comprehensive judgment—whether the button text and adjacent descriptions contain keywords such as "submit," "confirm," "save," "agree," "place order," "create," "send," "publish," "delete," "authorize," or "pay"; whether the page has collected complete business objects (e.g., amount, order, recipient, address, approval comments); whether there is dirty form data before clicking and whether the application history shows that "next" is automatically saved; whether the button has caused the generation of order numbers, work order numbers, message IDs, or approval serial numbers during historical page transitions; and whether observable network requests point to interfaces such as create / submit / pay / send / delete / authorize. When any of the above high-risk evidence reaches the threshold, the system upgrades the action from E2 to E3, prohibiting automatic execution in controlled real-world probing, thereby avoiding risk misjudgment due to ambiguous interface text. If a prediction error occurs during the actual execution phase and produces real side effects, the system will perform tiered compensation: For reversible states such as local input, drafts, favorites, and filter conditions, it will perform actions such as undo, clear, return, or restore snapshots; for compensable transactions (such as saved drafts, created but not submitted orders, and sent revocable messages), it will immediately call compensation actions such as undo / delete drafts / cancel orders / retract messages and record the results; for irreversible or external side effects (such as payments, formal submissions, authorizations, and public releases), it will immediately stop subsequent operations, freeze tokens, prompt users for manual handling, and write the reason for the misjudgment, page status, action evidence, and optional remedial paths into the audit chain.

[0055] The sandbox pre-simulation state machine can employ at least one of the following pre-simulation modes—virtual pre-simulation, shadow page pre-simulation, and controlled real-world probing pre-simulation—to perform secure pre-simulation of candidate operation sequences. In one implementation, the selection strategy for the three pre-simulation modes is as follows: if a page flow model or structured tool description exists, virtual pre-simulation is prioritized; if the target application provides a sandbox page, a copy of test data, a draft container, or an enterprise-controlled environment, shadow page pre-simulation is used; if none of the above conditions are met, controlled real-world probing pre-simulation is used. The system can automatically select or combine the above pre-simulation modes based on the target application type, environmental conditions, and security policies of the current task.

[0056] The virtual pre-simulation simulates page state changes based on UI semantic graphs and page state transition models. That is, without touching the real interface, the system simulates the page state changes after each action is executed, based on the node relationships, edge jump relationships in the UI semantic graph, and preset page state transition rules. This allows the system to obtain the expected page state without affecting any real business operations. Virtual pre-simulation is suitable for scenarios where the page flow is known and subsequent states can be predicted without rendering the real interface. Its advantage lies in completely isolating the real business environment, resulting in zero risk.

[0057] The shadow page pre-rendering executes a sequence of candidate operations in an isolated environment. Specifically, when the target application provides a sandbox page, a copy of test data, a draft container, or an enterprise-controlled environment, the system executes the sequence of candidate operations in that isolated environment to obtain the actual page rendering result and the next page structure, but without writing it into the production business state. Shadow page pre-rendering is suitable for proprietary applications or enterprise application scenarios. Its advantage lies in obtaining highly realistic page rendering results while ensuring that production data is not contaminated.

[0058] The controlled real-world probing rehearsal executes only actions belonging to the safe type set within a pre-defined, restricted real-world page to probe the state of subsequent pages. Specifically, when a shadow environment is unavailable (e.g., for third-party applications), the system executes actions belonging to the safe type set (i.e., E0, E1, or E2 type actions) from the candidate operation sequence on the real mobile page, but does not execute any actions belonging to the risk type set. By executing these low-risk actions, page jumps or state changes are triggered, thereby collecting real interface information for subsequent pages. In third-party application scenarios without a shadow environment, the system does not use real-world probing as a free exploration method by default. Instead, it first performs a virtual rehearsal based on historical page models, the current UI semantic graph, public page structures, user local historical paths, rule bases, and known application flows. Only when the information of the next page is necessary to complete the task, and the action to be executed is evaluated as low-risk (E0 / E1 / E2), reversible, or without external side effects, is it allowed to advance the real page once or a limited number of times with a probe token. The core constraints of the controlled real-world detection simulation are "only actions belonging to the security type set are executed" and "preset restrictions." This means the system strictly limits detection behavior to a secure range through a token mechanism and hard boundaries. These hard boundaries include: maximum number of detection steps, maximum page depth, maximum dwell time, minimum click interval, prohibition of bypassing CAPTCHAs, prohibition of simulating biometrics / payment passwords / SMS verification codes, and prohibition of accessing E3 / E4 actions such as payment submission / sending / deletion / authorization / public posting. Furthermore, if risk control prompts, abnormal login alerts, frequent operation prompts, or new permission requests appear during the detection process, the system immediately stops and enters manual control.

[0059] In one implementation, for third-party application scenarios, the system performs an AntiAutomationRisk assessment before executing a controlled real-world probe rehearsal. The assessment factors for AntiAutomationRisk include: application type and sensitivity level (e.g., financial applications, social applications, and e-commerce applications have higher sensitivity levels), whether the account is a real personal account (test accounts or virtual accounts have higher risk), whether the page involves login / payment / financial / social media dissemination, whether CAPTCHA / slider / face / SMS verification / risk control prompts appear, operation frequency per unit time, depth of continuous page jumps, whether it requires reading or inputting sensitive credentials (e.g., ID card number, bank card number, payment password), and whether it triggers system security pop-ups. Each assessment factor is weighted according to a preset risk value. If the overall risk value exceeds a preset threshold, the system stops the real-world probe, switches to a virtual rehearsal, requests the user to manually advance one page, or requires the user to switch to a test account or enterprise sandbox environment and retry. Simultaneously, the system records the action, reason, page summary, and reason for stopping each probe in the audit chain to prove that the system has not circumvented the security mechanisms of the third-party application.

[0060] The sandbox rehearsal is defined by the following boundaries: Low-risk, reversible, or side-effect-free actions are allowed to proceed in a controlled manner to discover subsequent page states and confirmation nodes; while actions that would have external business consequences (such as network submissions, fund changes, permission modifications, external dissemination, etc.) are subject to truncation, registration, and confirmation requirements, without executing real business logic. This rehearsal is not a complete execution of a task in a real business environment, nor is it merely a model-generated natural language judgment. Instead, it achieves a balance between security and information integrity during the rehearsal phase through a differentiated processing strategy of "controlled execution of safe actions for detection, and truncation and registration of risky actions pending confirmation."

[0061] Therefore, the three pre-simulation modes are suitable for different scenarios: virtual pre-simulation is suitable for scenarios where the process is known and the risk is zero; shadow page pre-simulation is suitable for self-developed application scenarios with an isolated environment and high realism; and controlled real detection pre-simulation is suitable for third-party application scenarios without a shadow environment and can obtain subsequent page information under controllable risk. The system can automatically select or combine the above pre-simulation modes according to the target application type, environmental conditions and security policies of the current task, so as to achieve safe pre-simulation of candidate operation chains in various application scenarios and achieve a balance between security and information integrity in the pre-simulation stage.

[0062] In one embodiment, the recording of the corresponding audit event generates a replayable audit chain, including: S1210. Record the currently completed action to generate the audit event; S1220. Obtain the hash value of the previous audit event, write it into the audit event, and perform hash calculation on the audit event to which the hash value of the previous audit event is written, to obtain the hash value of the audit event. S1230. The audit events with the hash values ​​are linked sequentially according to the execution order of the actions. When the audit events corresponding to all actions are linked, the replayable audit chain is obtained.

[0063] In this embodiment, after each action is completed, the system immediately records the corresponding audit event to form traceable execution evidence. Specifically, the system first records the currently completed action and generates the audit event. The audit event is a structured data object containing an event identifier, task identifier, operation sequence identifier, token identifier, timestamp, action type (e.g., click, input, swipe), target object identifier, action execution result (success or failure), and summary information of the current real UI state (e.g., page identifier, key control text, and business object value). In one implementation, the audit event also includes the expected page state hash value, the real page state hash value, the UI semantic graph hash value, difference score, risk level, the reason the model selected the action, the token verification result, and the user's secondary confirmation record, etc. Thus, the execution evidence of each action is solidified upon completion, avoiding the problem of being unable to reconstruct the execution scene due to the loss of the execution state.

[0064] When generating the audit event, the system obtains the hash value of the previous audit event and writes it into the current audit event. Specifically, for the first audit event (i.e., the audit event corresponding to the first action), the hash value of its previous audit event can be set to a preset initial value (e.g., an all-zero hash value). For each subsequent audit event, the system reads the hash value of the most recent audit event from the existing replayable audit chain and writes it as the "previous audit event hash value" field into the current audit event. Then, the system performs a hash calculation on the audit event for which the previous audit event hash value has been written, that is, it takes all fields of the current audit event (including the previous audit event hash value field) as input and performs a calculation using a preset hash algorithm (e.g., SHA-256) to obtain the hash value of the audit event. The hash value serves as a unique digital fingerprint of the current audit event, uniquely representing all the content of the audit event. Any modification to the content of the audit event will cause an unhideable change in its hash value. Since the current audit event contains the hash value of the previous audit event, the hash value of the current audit event indirectly depends on the content of all previous audit events, thus forming an interlocking hash chain structure.

[0065] Finally, the system links the audit events with the hash values ​​sequentially according to the execution order of the actions—that is, according to the execution order of each action in the operation sequence, the corresponding generated audit events are added one by one to the end of the audit event sequence. When all audit events corresponding to all actions in the operation sequence have been generated and linked, the replayable audit chain is obtained. Each audit event in the replayable audit chain contains the hash value of the previous audit event and its own hash value. If the content of any audit event is tampered with, its own hash value will change, resulting in a mismatch between the hash value of the previous audit event stored in subsequent audit events and the hash value of the previous audit event, thus enabling the tampering behavior to be quickly detected and located. In one embodiment, after the task is completed, the system performs an integrity check on the entire replayable audit chain, that is, it checks from the beginning whether the hash value of each audit event matches its content, and whether the hash value of the previous audit event stored in each audit event is consistent with the hash value of the previous audit event; if the check fails, the replayable audit chain is marked as "tampered with" and an abnormal event is recorded. In one implementation, the system supports uploading replayable audit chains to an enterprise audit server for centralized storage and management, and provides audit report export functionality. When storing the audit chain, the system can employ differentiated storage strategies based on task type and user configuration: for ordinary tasks, it can save control tree summaries and low-resolution screenshot thumbnails to reduce storage costs; for enterprise scenarios, it can save the complete evidence chain and upload it to the enterprise audit server to meet compliance requirements; for pages containing sensitive content, it can partially mask screenshots, retaining only the control structure and action summaries to protect user privacy. Thus, the replayable audit chain not only completely records the operation sequence, interface state, permission verification results, and decision-making reasons throughout the agent's execution process, but also ensures the audit records are tamper-proof through a hash chain structure, providing reliable technical support for accurate replay of the execution process, locating the causes of anomalies, and meeting enterprise compliance audit requirements.

[0066] In one embodiment, after pausing execution and triggering a re-enactment, the method further includes: S1310. Obtain the execution status information when paused and generate a manual takeover interface, wherein the manual takeover interface includes prompts for undone operations and corresponding undone entry points; S1320. In response to the user-triggered undo operation, perform the corresponding rollback process; or in response to the user-selected re-preview instruction, re-execute the pre-preview stage based on the current real UI state. If the preview passes, output the updated operation sequence, minimum permission token, and expected page state sequence, and continue execution from the breakpoint.

[0067] In this embodiment, when execution is paused and a re-render is triggered due to reasons such as discrepancy verification failure, token verification failure, the appearance of a verification code or system permission pop-up, user refusal of high-risk confirmation, task exceeding the number of steps, time or cost limits, or audit chain writing failure during the actual execution phase, the system first obtains the execution status information at the time of the pause and generates a manual takeover interface. The execution status information includes at least: a list of completed actions and their corresponding execution results (including whether each executed action was successful, confirmed, or revocable), a snapshot of the current real page and the control tree, the next action originally planned by the agent and its target object, the specific reason for the pause (e.g., "payment amount changed from 100 yuan to 1000 yuan, discrepancy verification failed" or "system permission pop-up detected, user intervention required"), and a list of revocable operations in the currently executed actions and their corresponding revocation methods. Based on the above execution status information, the system generates a manual takeover interface and displays it to the user. The manual takeover interface includes prompts for revocable operations and corresponding revocation entries. Specifically, for completed and reversible operations (e.g., "Message sent to Finance Xiao Li", "Draft saved locally", "Added to favorites"), the system displays a description of each reversible operation in a list or card format on the interface, and provides a corresponding interactive reversal entry for each operation (e.g., "Recall Message", "Delete Draft", or "Remove from Favorites" button). For completed but irreversible operations (e.g., payment completed, formal submission completed), the system clearly marks them as irreversible on the takeover interface and alerts the user, without providing a reversal entry, only recording the risk and reminding the user. Furthermore, the manual takeover interface also displays the expected reversal effect of reversible operations (e.g., "After recalling the message, the other party will not see the message" or "After deleting the draft, the filled content will be cleared and cannot be recovered") to help users make informed decisions. Thus, users can fully understand the current execution progress, the reason for the pause, the reversibility status of the executed operations, and the remedial measures that can be taken on the takeover interface, reducing user confusion and operational burden caused by abnormal interruptions of the intelligent agent.

[0068] In response to a user-triggered undo operation, the system executes the corresponding rollback process. Specifically, when a user clicks the undo entry for a reversible operation, the system invokes the corresponding rollback capability—for example, calling the "recall message" interface provided by the instant messaging application, performing the "delete draft" interface operation, or calling the "remove favorites" function—and updates the interface state after the rollback is complete, providing feedback to the user on the rollback execution result (e.g., "message has been recalled" or "draft has been deleted"). For locally reversible operations (e.g., form filling or option selection), the system can achieve rollback by restoring the state snapshot recorded during the pre-launch phase or performing the reverse operation (e.g., clearing the input box or deselecting), and informs the user of the rolled-back status. If an anomaly occurs during the rollback process (e.g., message recall timeout or insufficient recall permissions), the system displays the anomaly information in the takeover interface and suggests that the user handle it manually. Thus, users can proactively intervene and undo risky operations that have already been executed when the agent deviates from its intended purpose, reducing losses caused by misoperation.

[0069] On the other hand, in response to the user's re-preview instruction selected in the manual takeover interface, the system re-executes the pre-preview stage based on the current real UI state. Specifically, the system re-acquires the interface information of the current foreground application and constructs a UI semantic graph. Using the current real page as the starting state and the completion conditions in the original target task as the target state, it regenerates candidate operation sequences and inputs these sequences into the sandbox pre-preview state machine for security pre-preview. If the pre-preview passes, the system outputs the updated operation sequence, the updated minimum privilege token, and the updated expected page state sequence, and resumes execution from the point of interruption—that is, starting from the actions that were not executed when paused, and executing subsequent actions sequentially according to the updated operation sequence, without requiring the user to restart the entire task from scratch. If the re-preview fails (e.g., the current interface state cannot complete the target task or there is an unacceptable risk), the system remains on the manual takeover interface and prompts the user to manually complete the remaining operations. Therefore, when changes in the execution environment cause a deviation between the pre-simulation state and the real state, the system can re-evaluate and authorize based on the current real state, adapting to dynamic changes in the interface with minimal execution cost. This avoids the inefficiency of traditional solutions where users have to completely restart the task once an error occurs. At the same time, it ensures that the re-executed operation sequence still undergoes complete pre-simulation verification and token authorization, without skipping any security steps.

[0070] In one embodiment, verifying the difference between the current actual UI state and the corresponding expected page state includes: S1410. Obtain the difference information between the current actual UI state and the corresponding expected page state in multiple preset dimensions; S1420. Calculate the overall difference value based on the weights corresponding to each preset dimension; S1430. If the overall difference value does not exceed the preset threshold and no difference appears in the preset key dimensions, the difference verification is determined to be passed; if any key preset dimension shows a difference, the difference verification is determined to be failed.

[0071] In this embodiment, during the actual execution phase, after completing the user confirmation judgment before each action is executed, the system performs a state difference verification to verify whether the current actual UI state still meets the expectations of the pre-rehearsal phase. Specifically, the system first obtains the difference information between the current actual UI state and the corresponding expected page state in multiple preset dimensions. The multiple preset dimensions include, but are not limited to: layout difference (LayoutDelta), used to characterize whether the position, size, and hierarchical structure of controls have changed; text difference (TextDelta), used to characterize whether the displayed text of buttons, labels, field prompts, etc., has changed; business object difference (ObjectDelta), used to characterize whether key business objects (such as recipients, product specifications, shipping addresses, approval objects) have changed; risk difference (RiskDelta), used to characterize whether the risk level or business semantics of controls have changed (e.g., "Save Draft" becomes "Publish Now"); amount difference (AmountDelta), used to characterize whether key values ​​such as amount, quantity, and date have changed; and pop-up difference (DialogDelta), used to characterize whether unexpected pop-ups or system prompts have appeared. In one implementation, the system calculates the degree of difference for each preset dimension. The degree of difference can be quantified based on algorithms such as control matching rate (e.g., the proportion of the number of controls in the current interface that match the pre-show state to the total number of controls in the pre-show state), text editing distance (e.g., Levenshtein distance), business object identification comparison (e.g., whether the contact ID and order number are consistent), risk level difference, relative change rate of values, and pop-up type matching results to obtain the difference value for each dimension.

[0072] Subsequently, the system calculates the comprehensive difference value based on the weights corresponding to each preset dimension. The comprehensive difference value is a weighted sum of the difference values ​​for each dimension, and its calculation formula is: DriftScore = α·LayoutDelta + β·TextDelta + γ·ObjectDelta + δ·RiskDelta + ε·AmountDelta + ζ·DialogDelta, where α, β, γ, δ, ε, and ζ are the weight coefficients corresponding to each preset dimension. In one implementation, the weight coefficients are dynamically configured—different weight allocation strategies are used for different task types, different risk levels, and different page roles. For example, for payment tasks, the weights for amount difference and business object difference are configured to be much higher than other dimensions; for news browsing tasks, the weights for text difference and layout difference are relatively high, while the weight for amount difference is zero. In one implementation, during the task access phase, the system automatically selects the corresponding weight configuration strategy based on the nature of the target task and writes the difference verification strategy parameters (DriftPolicy) adopted by the current task into the least privilege token. These difference verification strategy parameters include at least: task type, risk level, page type, object sensitivity, field tolerance, user tolerance, and enterprise policy. During actual execution, the system loads the corresponding weights and thresholds from the difference verification strategy parameters based on the business risk of the current step—high-risk steps such as payment, transfer, order placement, sending, deletion, authorization, and approval use low-tolerance thresholds; low-risk steps such as browsing news, ordinary search, and viewing stateless pages use higher-tolerance thresholds. Therefore, the system can flexibly adjust the sensitivity of difference verification according to the task type and risk scenario, avoiding overly conservative thresholds in low-risk scenarios leading to frequent false alarms, or overly lenient thresholds in high-risk scenarios leading to missed risks.

[0073] After obtaining the overall difference value, the system executes the judgment logic: if the overall difference value does not exceed the preset threshold and no difference appears in the preset key dimensions, the difference verification is deemed to have passed; if any key preset dimension shows a difference, the difference verification is deemed to have failed. Specifically, the system sets sub-thresholds and hard blocking conditions for the difference items, rather than relying solely on the total score of the overall difference value for judgment. The preset key dimensions refer to dimensions that have a decisive impact on the security of the current task. They are dynamically determined by the system according to the task type or preset by the user / enterprise administrator. For example, for payment tasks, the amount, recipient, and product specifications are key dimensions, and even a change of 0.01 yuan in the amount will directly trigger a freeze or user confirmation; for enterprise approval tasks, the approval amount, approval object, and approval number are key dimensions, and any change will result in a failure of the verification and trigger a re-rehearsal; for ordinary form filling tasks, the key dimensions can be configured as whether the business object (e.g., applicant, reason) has changed. In one implementation, the system allows users or enterprise administrators to preset difference tolerance policies for specific applications, pages, or tasks through policy configuration files. For example, a policy could be set to "zero tolerance for approved amounts, confirmation required for adding attachments to the list, and minor adjustments to page text can be ignored." This policy is issued along with the least privilege token and takes effect during the difference verification phase. Furthermore, even if a change does not fall into a critical dimension and the overall difference value does not exceed a threshold, the system can still decide whether to issue a lightweight notification to the user based on the actual risk level of the change. For example, a notification could be sent stating "The page layout has changed, but the target action is unaffected," to keep the user informed.

[0074] Therefore, by acquiring and weighting multi-dimensional difference information, the system achieves a refined quantitative assessment of the difference between the actual UI state and the expected page state. By setting a hard blocking mechanism for key dimensions, changes in core business data such as amount and recipient can trigger security interception directly, independent of the comprehensive threshold. The configuration mechanism that allows weights and thresholds to be dynamically adjusted according to task type ensures the adaptability of the verification strategy in different task scenarios. The dual judgment logic combining comprehensive difference value verification and key dimension verification allows the system to tolerate minor UI drifts that do not affect task security and correctness (such as changes in ad placements or timestamp updates), while maintaining zero tolerance for any changes in core business data. This ensures the continuity of automated execution while ensuring that each step of the agent's execution does not deviate from the security boundaries verified in the pre-rehearsal stage.

[0075] Figure 2 A schematic diagram of the terminal intelligent agent operation safety control device provided in an embodiment of the present invention is shown. This device, as... Figure 2 As shown, it includes: The data acquisition module 310 is used to acquire, based on the target task triggered by the user, the operation sequence, the least privilege token, and the expected page state sequence generated through the pre-rehearsal stage; wherein, the operation sequence includes at least one action, and the least privilege token at least limits the scope of operations allowed to be performed in the target task, the validity period, and the types of actions that require user confirmation. The execution verification module 320 is used to execute each action in the operation sequence in sequence, including determining whether the target object of the action is within the scope of the operation object limited by the least privilege token; wherein, if the target object of the action is within the scope of the operation object and the type of the action is an action type that requires user confirmation, then the execution is paused and a secondary confirmation operation is initiated to the user; if the type of the action is not an action type that requires user confirmation or has already obtained secondary confirmation from the user, then the difference between the current real UI state and the corresponding expected page state is verified. The result generation module 330 is used to pause execution and trigger a re-rehearsal if the difference verification fails; if the difference verification passes, the action is executed and the corresponding audit event is recorded to generate a replayable audit chain.

[0076] Figure 3 The diagram shows a structural schematic of a computer device provided in an embodiment of the present invention. The specific embodiments of the present invention do not limit the specific implementation of the computer device.

[0077] like Figure 3 As shown, the computer device may include: a processor 402, a communications interface 404, a memory 406, and a communications bus 408.

[0078] The processor 402, communication interface 404, and memory 406 communicate with each other via communication bus 408. Communication interface 404 is used to communicate with other network elements such as clients or other servers. The processor 402 executes program 410, specifically performing the relevant steps described in the embodiment of the terminal intelligent agent operation security control method.

[0079] Specifically, program 410 may include program code, which includes computer-executable instructions.

[0080] Processor 402 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The computer device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0081] Memory 406 is used to store program 410. Memory 406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0082] Specifically, program 410 can be called by processor 402 to cause the computer device to perform the relevant steps in the embodiment of the terminal intelligent agent operation security control method.

[0083] This invention provides a computer-readable storage medium storing at least one executable instruction that, when executed on a computer device, causes the computer device to perform the terminal intelligent agent operation security control method in any of the above method embodiments.

[0084] This invention provides a computer program that can be invoked by a processor to cause a computer device to execute the terminal intelligent agent operation security control method in any of the above method embodiments.

[0085] This invention provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions, which, when executed on a computer, cause the computer to perform the terminal intelligent agent operation security control method in any of the above method embodiments.

[0086] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of the present invention are not directed to any particular programming language. It should be understood that the content of the invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the invention.

[0087] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0088] Similarly, it should be understood that, in order to streamline the invention and aid in understanding one or more of the various aspects of the invention, features of the embodiments of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of the invention. However, this disclosure should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim.

[0089] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0090] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A method for safe operation control of a terminal intelligent agent, characterized in that, include: Based on the target task triggered by the user, obtain the operation sequence, the least privilege token, and the expected page state sequence generated through the pre-rehearsal stage; wherein, the operation sequence includes at least one action, and the least privilege token at least limits the scope of operations allowed to be performed in the target task, the validity period, and the types of actions that require user confirmation; Each action in the operation sequence is executed sequentially, including determining whether the target object of the action is within the scope of the operation object limited by the least privilege token; wherein, if the target object of the action is within the scope of the operation object and the type of the action is an action type that requires user confirmation, then the execution is paused and a secondary confirmation operation is initiated to the user; if the type of the action is not an action type that requires user confirmation or has already obtained secondary confirmation from the user, then the difference between the current real UI state and the corresponding expected page state is verified. If the difference verification fails, execution is paused and a replay is triggered; if the difference verification passes, the action is executed, the corresponding audit event is recorded, and a replayable audit chain is generated. The pre-rehearsal stage includes: Obtain the interface information of the current foreground application on the mobile terminal, identify interactive control nodes from the interface information, and obtain a set of control nodes; Based on the control node set, a UI semantic graph is constructed, and the nodes in the UI semantic graph have business semantics and operation attribute information; Based on the target task triggered by the user and the UI semantic graph, a candidate operation sequence based on the target task is generated; The candidate operation sequence is input into the sandbox pre-simulation state machine, and each action contained in the candidate operation sequence is processed in sequence. For actions whose operation attribute information is a set of security types, the expected page state after execution is collected. For actions whose operation attribute information is a set of risk types, no real business action is executed, and they are determined to be action types that require user confirmation. The expected page states collected through the pre-rehearsal and the action types identified as requiring user confirmation are merged into the pre-rehearsal result; based on the pre-rehearsal result, the operation sequence, the least privilege token, and the expected page state sequence uniquely bound to the target task are generated.

2. The method according to claim 1, characterized in that, The step of obtaining the interface information of the current foreground application on the mobile terminal, identifying interactive control nodes from the interface information, and obtaining a set of control nodes includes: Obtain the interface information, which includes a control tree, screenshots, and accessibility node information; Interactive control nodes are extracted from the control tree and the accessibility node information to generate an initial set of control nodes; Visual recognition is performed on the screenshot, including OCR text recognition and icon semantic recognition; The results of the visual recognition are fused with the initial set of control nodes. For nodes that already exist in the initial set of control nodes, the recognition results are used to supplement or correct the corresponding operation attribute information. For nodes that are recognized but do not exist in the initial set of control nodes, new control nodes are created according to the corresponding node position and interaction type. The set of control nodes obtained through fusion will be used as the enhanced set of control nodes.

3. The method according to claim 1, characterized in that, The set of security types includes actions that have no external side effects or are reversible; the set of risk types includes actions that generate external network requests, fund changes, or permission changes. The sandbox pre-simulation state machine employs at least one of the following pre-simulation modes: virtual pre-simulation, shadow page pre-simulation, and controlled real-world detection pre-simulation. The virtual pre-simulation simulates page state changes based on a UI semantic graph and a page state transition model. The shadow page pre-simulation executes candidate operation sequences in an isolated environment. The controlled real-world detection pre-simulation executes only actions belonging to a set of safe types on a preset, restricted real page to detect subsequent page states.

4. The method according to claim 1, characterized in that, The audit events corresponding to the records generate a replayable audit chain, including: The audit event is generated by recording the currently completed action. Obtain the hash value of the previous audit event, write it into the audit event, and perform a hash calculation on the audit event to which the hash value of the previous audit event was written, to obtain the hash value of the audit event. The audit events with the hash values ​​are linked sequentially according to the execution order of the actions. Once the audit events corresponding to all actions are linked, the replayable audit chain is obtained.

5. The method according to claim 1, characterized in that, After pausing execution and triggering a re-render, the process also includes: Obtain the execution status information at the time of pause and generate a manual takeover interface, wherein the manual takeover interface includes prompts for undone operations and corresponding undone entry points; In response to a user-triggered undo operation, execute the corresponding rollback process; or in response to a user-selected re-preview instruction, re-execute the pre-preview stage based on the current real UI state. If the preview passes, output the updated operation sequence, the least privilege token, and the expected page state sequence, and continue execution from the point of interruption.

6. The method according to claim 1, characterized in that, The verification of the difference between the current actual UI state and the corresponding expected page state includes: Obtain information on the differences between the current actual UI state and the corresponding expected page state across multiple preset dimensions; Calculate the overall difference value based on the weights corresponding to each preset dimension; If the overall difference value does not exceed the preset threshold and no difference appears in the preset key dimensions, the difference verification is deemed to have passed; if any preset key dimension shows a difference, the difference verification is deemed to have failed.

7. A terminal intelligent agent operation safety control device, characterized in that, include: The data acquisition module is used to acquire, based on the target task triggered by the user, the operation sequence, the least privilege token, and the expected page state sequence generated through the pre-rehearsal stage; wherein, the operation sequence includes at least one action, and the least privilege token at least limits the scope of operations allowed to be executed in the current target task, the validity period, and the types of actions requiring user confirmation; wherein, the pre-rehearsal stage includes: Obtain the interface information of the current foreground application on the mobile terminal, identify interactive control nodes from the interface information, and obtain a set of control nodes; Based on the control node set, a UI semantic graph is constructed, and the nodes in the UI semantic graph have business semantics and operation attribute information; Based on the target task triggered by the user and the UI semantic graph, a candidate operation sequence based on the target task is generated; The candidate operation sequence is input into the sandbox pre-simulation state machine, and each action contained in the candidate operation sequence is processed in sequence. For actions whose operation attribute information is a set of security types, the expected page state after execution is collected. For actions whose operation attribute information is a set of risk types, no real business action is executed, and they are determined to be action types that require user confirmation. The expected page states collected through the pre-rehearsal and the action types identified as requiring user confirmation are merged into the pre-rehearsal result; based on the pre-rehearsal result, the operation sequence, the least privilege token, and the expected page state sequence uniquely bound to the target task are generated; The execution verification module is used to execute each action in the operation sequence in sequence, including determining whether the target object of the action is within the scope of the operation object limited by the least privilege token; wherein, if the target object of the action is within the scope of the operation object and the type of the action is an action type that requires user confirmation, then the execution is paused and a secondary confirmation operation is initiated to the user; if the type of the action is not an action type that requires user confirmation or has already obtained secondary confirmation from the user, then the difference between the current real UI state and the corresponding expected page state is verified. The result generation module is used to pause execution and trigger a re-run if the difference verification fails; if the difference verification passes, the action is executed and the corresponding audit event is recorded to generate a replayable audit chain.

8. A computer device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction, which causes the processor to perform the operation of the terminal intelligent agent operation security control method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one executable instruction, which, when executed on a computer device, causes the computer device to perform the operation of the terminal intelligent agent operation security control method as described in any one of claims 1-6.

Images