AI agent desktop task execution system and method based on isolated graphics execution

Abstract

The application provides an AI agent desktop task execution system and method based on isolated graphics execution, which comprises a task management module, a tool capability index module, a file authorization gateway module, an application starting agent module, a graphics operation adaptation module, an audit module, a result processing module and a result feedback module; all the modules jointly constitute the functional logic subject of the system resident process agent-workspace. The application generates a task level strategy configuration file for each task, creates an independent execution environment, limits the AI to only access authorized files and only start authorized applications, and forms result proof information, thereby improving the credibility and auditability of the operation result of the AI agent.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to an AI intelligent agent desktop task execution system and method based on isolated graphics execution. Background Technology

[0002] In desktop operating systems, users typically access applications through a main desktop session, such as opening office software, file managers, and browsers. With the increasing use of AI agents in desktop operating systems, users expect AI to assist in tasks such as organizing spreadsheets, editing documents, exporting PDFs, and processing files. For software with standardized interfaces, when executing user tasks, AI agents can prioritize calling application capabilities through standardized interfaces, such as MCP, Connector, plugin interfaces, or application open APIs.

[0003] However, in real-world scenarios, many office software programs, industry-specific software, or graphical tools lack complete standardized interfaces and still require operation through a GUI. This means that AI still needs to interact with windows, menus, buttons, and dialog boxes just like a user. Current methods for addressing these scenarios typically involve having the AI ​​directly operate the user's current desktop or performing automated operations within a regular sandbox or remote desktop. While these methods can accomplish some tasks, they easily interrupt the user's current work, affecting normal use. The AI ​​may also access unauthorized windows, files, or private data, making it difficult for the user to determine the reliability of the final generated file. Simply placing the AI ​​in a regular isolated desktop or sandbox, while restricting its operating environment, usually only addresses the location of execution and doesn't prove how the final file was generated, whether unauthorized resources were accessed, or whether any abnormal operations were rejected. For example, it's difficult for users to confirm which input files the result file was based on, which applications were launched during the AI ​​process, which files were accessed, whether unauthorized resources were attempted, and whether the result file was tampered with. Summary of the Invention

[0004] The purpose of this invention is to provide an AI agent desktop task execution system and method based on isolated graphical execution. It generates a task-level policy configuration file for each task, creates an independent execution environment, restricts the AI ​​to only access authorized files and launch authorized applications, and generates result proof information, thereby improving the credibility and auditability of the AI ​​agent's operation results.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows: An AI agent desktop task execution system based on isolated graphical execution includes a task management module, a tool capability index module, a file authorization gateway module, an application launch agent module, a graphical operation adaptation module, an audit module, a result processing module, and a result feedback module. All modules together constitute the functional logic of the system's resident process, agent-workspaced. agent-workspaced receives user requests and creates AI agent tasks through the task management module; it determines whether the AI ​​agent task can be completed through a standardized interface using the capability index of local tools built by the tool capability index module; if not, it proceeds to isolated graphical execution; it constructs an isolated environment and access control for the isolated graphical interface through the file authorization gateway module and the application launch agent module; it obtains the isolated graphical interface state and injects operation events through the graphical operation adaptation module; it records operation events and generates a hash-based anti-tampering audit event chain through the audit module; it calculates the hash of the AI ​​agent task's result file through the result processing module and generates result proof information; and it displays the AI ​​agent task result to the user, allows confirmation of the feedback result, or allows rejection of the feedback result through the result feedback module.

[0006] Furthermore, the task management module generates a unique task identifier (task_id) for each AI agent task, serving as a unified index for the execution process of the isolated graphical interface.

[0007] Furthermore, the tool capability index module loads the system's local tool capability description file, parses the capability, and builds a capability index. Before loading, it first creates a ToolCapability structure and a ToolRegistry structure in memory. The ToolCapability structure is used to describe the capability information of a single tool, and the ToolRegistry structure is used to store the capability information of all loaded tools.

[0008] Furthermore, the file authorization gateway module controls the file access scope of the AI ​​agent and maps authorized files to isolated directories.

[0009] Furthermore, the application launch agent module controls the AI ​​agent to launch a graphical application authorized by a task-level policy configuration file. The task-level policy configuration file is generated by agent-workspaced based on the AI ​​agent's task, user authorization, lack of local tool capabilities, and default policy template.

[0010] In another aspect, the present invention also proposes a method for using the aforementioned AI intelligent agent desktop task execution system based on isolated graphics execution, comprising: S1. The system resident process agent-workspaced connects to D-Bus and registers the D-Bus interfaces of the corresponding task management module, application startup agent module, file authorization gateway module, audit module, result processing module, and result return module. S2 and agent-workspaced load local tool capability description files through the tool capability index module, parse the files, and build an in-memory capability index. S3. agent-workspaced receives the user's AI agent task request through the task management module, creates the AI ​​agent task and matches it according to the capability index. If the match is successful, the task is completed through the standardized interface; otherwise, it proceeds to the isolated graphical execution process in step S4. S4 and agent-workspaced create root and isolation directories for AI agent tasks and dynamically generate task-level policy configuration files that configure authorization files, authorized applications, output directories, and disabled items. S5. Map authorized files to isolated directories through the file authorization gateway module and establish mandatory access control rules; S6. Start an isolated Wayland standalone graphical session and launch the application through the application launch agent module; S7. Obtain the interface state of the Wayland independent graphical session through the graphical operation adaptation module and send it to the AI ​​agent, and inject the operation events of the AI ​​agent into the interface of the Wayland independent graphical session; record the operation events of the AI ​​agent through the audit module and generate a hash chain-based anti-tampering audit event chain; S8. After the task is completed, the result processing module verifies the result file path and type, calculates the hash, and generates result proof information; the result feedback module displays the result to the user and terminates the AI ​​agent task.

[0011] Furthermore, after the D-Bus interface is registered in step S1, agent-workspaced configures access control policies for each interface, creates a service working directory and a runtime directory, and loads the default policy template used to generate task-level policy configuration files.

[0012] Furthermore, in step S2, before loading the tool capability description file, agent-workspaced first creates a ToolCapability structure and a ToolRegistry structure in memory. The ToolCapability structure is used to describe the capability information of a single tool, and the ToolRegistry structure is used to store the capability information of all loaded tools.

[0013] Furthermore, in step S3, the task management module generates a unique task identifier, task_id, which serves as a unified index for subsequent steps.

[0014] Furthermore, the dynamic generation of the task-level policy configuration file in step S4 includes: A task-level policy configuration file is generated based on the AI ​​agent's task, user authorization information, missing information about local tool capabilities, and the default policy template.

[0015] Compared with the prior art, the present invention has the following significant advantages: This invention generates a task-level policy configuration file for each AI agent task, creates an independent execution environment, restricts the AI ​​to accessing only authorized files and launching only authorized applications, and generates result verification information. Through this invention, the AI ​​agent's task authorization, execution process, anomaly interception, result files, and user confirmation actions are bound together into a verifiable link, enabling the AI ​​agent's graphical task results to be traceable, verifiable, and auditable, thus improving the credibility and auditability of the AI ​​agent's graphical operation results. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the system structure of Embodiment 1 of the present invention; Figure 2 This is a flowchart of the method in Embodiment 2 of the present invention. Detailed Implementation

[0017] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0018] Example 1: In this embodiment, the main design concept of the AI ​​agent desktop task execution system based on isolated graphical execution is to prevent the AI ​​agent from directly taking over the user's main desktop. Instead, it creates an isolated graphical execution environment for the AI ​​agent that is bound to the current task. This isolated graphical execution environment includes an independent agent identity, an independent graphical session, an isolated file directory, an authorized input directory, a working directory, an output directory, and an audit directory.

[0019] like Figure 1 As shown, the AI ​​intelligent agent desktop task execution system based on isolated graphics execution includes a task management module, a tool capability index module, a file authorization gateway module, an application startup agent module, a graphics operation adaptation module, an audit module, a result processing module, and a result feedback module.

[0020] Task Management Module (Manager Module): Receives task submissions from user AI agents, generates task identifiers, and manages the task lifecycle; Tool Registry module: Loads the system's local tool capability description file, parses the standardized tool interface capability description, and determines whether the task can be completed through the standardized interface; The FileGateway module controls the file access scope of the AI ​​agent by mapping authorized files to isolated directories via bindmount. The AI ​​Agent can only access these authorized files and cannot directly access the user's entire directory. Application Launch Agent Module (AppBroker Module): Controls the AI ​​agent to launch graphical applications, allowing only applications authorized by the task-level policy configuration file to launch, and refusing to launch unauthorized applications and recording the reason for the refusal; The UI Adapter module (graphics operation adaptation module) acquires the interface state and injects mouse, keyboard, and other operation events within an isolated graphical session that isolates the graphical execution environment. This module only operates within the Agent's isolated graphical session and does not affect the user's main desktop. In other words, the AI ​​Agent's clicks, inputs, window switching, menu selections, and other operations will only occur within the isolated graphical execution environment, will not move the user's current mouse, and will not affect the windows currently in use on the user's main desktop.

[0021] The Audit module generates a hash-chain-based, tamper-proof audit event chain. An audit event is a structured operation record generated during the execution of a task by the AI ​​Agent, describing the time, subject, object, type, and result of a key operation. An audit event chain is a sequence of events formed by connecting multiple audit events from the same task in chronological order. To prevent tampering, this embodiment calculates the hash value of each audit event based on the hash value of the previous event and the content of the current event. Thus, if any audit event is modified, the hash values ​​of subsequent events will change, allowing the detection of audit record tampering. Through the audit event chain, the system can trace which files the AI ​​Agent accessed, which applications it launched, which graphical operations it performed, and which unauthorized requests were rejected by the system during task execution.

[0022] The Result Processing Module calculates the hash of the result file and generates result verification information. This verification information is a document generated by the system after the AI ​​Agent generates the result file, based on the task-level policy configuration file, input file information, audit event chain, and the result file hash. This verification information explains which task generated the result file, which input files were used, which applications were launched, what key operations occurred during execution, whether any unauthorized actions were rejected, what the result file hash is, and whether the result was delivered after user confirmation. The result verification information can be saved along with the result file, allowing users to verify the source and execution process of the result file upon receiving it from the AI ​​Agent.

[0023] The Handoff module displays the AI ​​agent's task results to the user, allowing them to confirm or refuse to submit the results.

[0024] The task-level policy configuration file refers to a policy configuration file dynamically generated by the system based on the user's AI agent task. This file is the permission control file corresponding to the current AI agent task, with the file format policy.json, and is stored in the execution directory of the corresponding task. The task-level policy configuration file describes which agent identity the AI ​​Agent is allowed to use, which input files it can access, which applications it can launch, which directory it can output results to, whether network access is allowed, whether clipboard sharing is allowed, which operation behaviors need to be logged, and whether the results need user confirmation before being returned. Subsequently, the file authorization gateway module, application launch agent module, graphical operation adaptation module, result processing module, and result return module will all read the task-level policy configuration file policy.json corresponding to this task and decide whether to allow or deny the relevant operations based on the policies contained therein.

[0025] All modules together constitute the functional logic core of the system's resident process, agent-workspaced. agent-workspaced is started by systemd after system startup and registers the D-Bus control interface during the initialization phase. This D-Bus control interface includes D-Bus interfaces for the corresponding task management module, application startup agent module, file authorization gateway module, auditing module, result processing module, and result feedback module. It is used to receive task requests, control application startup, manage file authorization, record audit events, and return execution results. After each AI agent task is created, agent-workspaced creates an isolated directory for task execution, an independent Agent identity, an independent graphical session, and mandatory access control rules based on the task-level policy configuration file corresponding to that task. It also creates a local Unix Domain Socket for the AI ​​agent task, allowing the graphical operation adaptation module to obtain the isolated graphical interface status and inject input events. Through this method, the AI ​​Agent does not directly access the user's home directory, directly call execve to start system applications, or directly write to the user's target directory. Instead, it completes task execution through the controlled interfaces created and managed by agent-workspaced.

[0026] This embodiment binds the AI ​​agent's task authorization, execution process, anomaly interception, result file, and user confirmation action into a verifiable link, enabling the AI ​​agent's graphical task results to be traceable, verifiable, and auditable, thereby improving the credibility and auditability of the AI ​​agent's graphical operation results.

[0027] Example 2: This embodiment describes the specific usage method and execution flow of the AI ​​intelligent agent desktop task execution system based on isolated graphics execution described in Embodiment 1, such as... Figure 2 As shown, it includes: Step 1: Start the agent-workspaced service and register the controlled interfaces.

[0028] After system startup, systemd starts the persistent system process agent-workspaced as a service. agent-workspaced enters the initialization process, first connecting to the system D-Bus and registering a service name; for example, the registered service name is org.openkylin.AgentWorkspace1. After completing the service name registration, agent-workspaced registers multiple D-Bus objects under that service name: a task management object for the corresponding task management module is registered on / org / openkylin / AgentWorkspace1 / Manager, providing SubmitTask, GetTaskStatus, and CancelTask ​​interfaces for receiving task requests, querying task status, and canceling tasks; an application launch agent object for the corresponding application launch agent module is registered on / org / openkylin / AgentWorkspace1 / AppBroker, providing LaunchApp and TerminateApp interfaces for controlling AI. The Agent starts or terminates the authorization application; registers the corresponding file authorization gateway object for the file authorization gateway module on / org / openkylin / AgentWorkspace1 / FileGateway, and provides PrepareMounts and CheckAccess interfaces for establishing authorized file mappings and verifying file access behavior; registers the corresponding audit object for the audit module on / org / openkylin / AgentWorkspace1 / Audit, and provides AppendEvent interface for unified writing of audit event chains; registers the corresponding result processing object for the result processing module on / org / openkylin / AgentWorkspace1 / Result, and provides Finalize interface for calculating hashes and generating result proof information after the result file is generated; registers the corresponding result return object for the result return module on / org / openkylin / AgentWorkspace1 / Handoff, and provides Prepare, Commit, and Reject interfaces for displaying results to users, confirming return results, or rejecting return results.

[0029] After the D-Bus interfaces are registered, agent-workspaced configures access control policies for each interface. Specifically, the Manager interface allows calls from the Agent frontend on the user's main desktop; the AppBroker, FileGateway, Result, and Handoff interfaces are only allowed to be called by the Agent runtime corresponding to the current task, the user's main desktop Agent frontend, or authorized calls from agent-workspaced; the Audit.AppendEvent interface is only allowed to be called by modules within agent-workspaced or modules authorized by the service. Ordinary Agent processes, user applications, and unauthorized processes cannot directly call this interface, nor can they directly modify the audit event files in the audit directory. Through this interface access control, the AI ​​Agent can only trigger audit events through controlled interfaces such as AppBroker, FileGateway, and ui.sock, and cannot forge or tamper with the audit event chain.

[0030] After configuring interface permissions, agent-workspaced creates the service working directory ` / var / lib / agent-workspace` and the runtime directory ` / run / agent-workspace`, and loads the default policy template `default-policy.json`. This default policy template stipulates that, unless explicitly authorized by the task-level policy configuration file, AIAgent by default prohibits access to the network, prohibits clipboard sharing, prohibits access to the user's full home directory, prohibits launching high-risk applications such as terminals, browsers, system settings, and password managers, and prohibits direct writing to the user's home directory. At this point, agent-workspaced has completed the initialization of basic services, controlled interfaces, and the default policy template.

[0031] Step 2: agent-workspaced loads the tool capability description file and builds the tool capability index.

[0032] After basic service initialization is complete, agent-workspaced calls `tool_registry_load()` in the tool capability index module to load local tool capability description files. These tool capability description files declare the standardized tool interface capabilities already provided by a particular application or system component, enabling agent-workspaced to determine before subsequent task execution whether a task can be completed through standardized interfaces such as MCP, Connector, plugin interfaces, or application open APIs. Tool capability description files are divided into two categories: system default description files and locally overriding description files. System default description files are stored in the ` / usr / share / agent-workspace / tools / ` directory. Files in this directory are installed or updated by system packages and are used to provide descriptions of application tool capabilities supported by default by the operating system. Locally overriding description files are stored in the ` / etc / agent-workspace / tools.d / ` directory. Files in this directory are generated by system administrators, device manufacturers, or upper-layer product configurations and are used to add tool capabilities, disable some default capabilities, or override system default descriptions. Since the ` / usr / share / agent-workspace / tools / ` directory may be updated during package upgrades, and the ` / etc / agent-workspace / tools.d / ` directory is used to store local policies and administrator configurations, when tool capability description files with the same `app_id` exist in both directories, agent-workspaced will use the description file in ` / etc / agent-workspace / tools.d / ` to override the default description file in ` / usr / share / agent-workspace / tools / `, thus ensuring that local configurations take precedence.

[0033] Before loading the tool capability description file, agent-workspaced first creates the ToolCapability structure and the ToolRegistry structure in memory.

[0034] ToolCapability describes the capabilities of a single tool. For example, its fields include app_id, interface_type, bus_name, object_path, method_list, supported_mime_types, capability_list, source_path, and priority. Here, app_id represents the application identifier corresponding to the tool; interface_type represents the interface type; bus_name and object_path represent the tool's calling address; method_list represents the list of methods provided by the tool; supported_mime_types represents the file types the tool supports; capability_list represents the set of capabilities supported by the tool; source_path indicates which configuration file the capability description originates from; and priority indicates the priority of the capability description, with the description file in / etc / agent-workspace / tools.d / having higher priority than the description file in / usr / share / agent-workspace / tools / .

[0035] The ToolRegistry stores capability information for all loaded tools. For example, its fields include tool_map, source_dirs, loaded_count, registry_generation, registry_hash, and registry_lock. Specifically, tool_map is a hash table with app_id as the key, and the values ​​are the corresponding ToolCapability objects; source_dirs records the directories of capability description files used in this loading, including / usr / share / agent-workspace / tools / and / etc / agent-workspace / tools.d / ; loaded_count records the number of tools successfully loaded; registry_generation records the version number of the tool capability index, incrementing with each reload; registry_hash records the digest value calculated from all current tool capability descriptions, used to determine if the tool capability index has changed; and registry_lock provides concurrent access protection when tasks query the tool capability index, preventing service reloading of capability description files from affecting the judgment of currently executing tasks.

[0036] When `tool_registry_load()` executes, it first scans the ` / usr / share / agent-workspace / tools / ` directory, reads the system's default tool capability description files, and parses each description file into a `ToolCapability` object. After parsing, `tool_registry_load()` checks the validity of the `app_id`, `interface_type`, `capability_list`, `method_list`, and `supported_mime_types` fields in the `ToolCapability` object. If a necessary field is missing or the field format does not meet the requirements, `agent-workspaced` will not add the tool capability description to the tool capability index and will log an initialization error. If the field check passes, `tool_registry_load()` inserts the `ToolCapability` object into `ToolRegistry.tool_map` using `ToolCapability.app_id` as the key. After the system's default description files are loaded, `tool_registry_load()` continues to scan the ` / etc / agent-workspace / tools.d / ` directory, reads the local overwrite description files, and similarly parses them into `ToolCapability` objects. If the same app_id does not already exist in ToolRegistry.tool_map, the ToolCapability object is directly inserted into tool_map. If the same app_id already exists in ToolRegistry.tool_map, the priority fields of the two objects are compared. If the local overriding description file has higher priority, the ToolCapability object parsed from / etc / agent-workspace / tools.d / is used to replace the original object. After replacement, the tool capabilities corresponding to the app_id in tool_map are based on the local overriding description file. After all description files are loaded, tool_registry_load() updates ToolRegistry.loaded_count, registry_generation, and registry_hash, and retains the ToolRegistry in the agent-workspaced process memory as a capability index for subsequent standardized tool interface capability judgment.

[0037] After the controlled interface registration and tool capability index are completed, agent-workspaced enters the event loop and continuously listens for task requests on D-Bus.

[0038] Step 3: The user submits a task, and the Manager module creates a task object and determines the capabilities of the standard tool interface.

[0039] After a user submits a task to the AI ​​agent on the main desktop, the user session-side Agent frontend calls the D-Bus interface `org.openkylin.AgentWorkspace1.Manager.SubmitTask` to send the task request to `agent-workspaced`. For example, the input parameters for `SubmitTask` include `user_id`, `task_prompt`, `input_files`, `target_apps`, and `output_type`. Here, `user_id` represents the user initiating the task, `task_prompt` represents the user's task description, `input_files` represents the list of input files authorized by the user, `target_apps` represents the target applications allowed by the user, and `output_type` represents the target output type. Upon receiving the `SubmitTask` call, the Manager interface generates a unique task identifier `task_id` and creates an `AgentTask` object. The `AgentTask` contains the fields `task_id`, `user_id`, `task_prompt`, `input_files`, `target_apps`, `output_type`, and `status`. After the task is created, `status` is set to `TASK_CREATED`. The task_id will subsequently serve as a unified index for generating task-level policy configuration files, creating isolated execution environments, file authorization mapping, application startup control, audit event logging, result proof generation, and result feedback.

[0040] Subsequently, agent-workspaced calls task_analyze() to parse the task description. For example, task_analyze() determines the target input files, target graphical applications, input file type set input_mime_types, target output type, and required capabilities set required_capabilities for the current task based on task_prompt, input_files, target_apps, and output_type. After the task analysis is complete, the parsing results are written to AgentTask.

[0041] After task parsing is complete, agent-workspaced calls tool_capability_check(task_id) to determine whether the standard tool interface can complete the current task. tool_capability_check() reads the candidate application list candidate_app_ids, the input file type set input_mime_types, and the task required capabilities set required_capabilities from AgentTask. Then, it iterates through candidate_app_ids and uses the app_id of each candidate application as the key to look up the corresponding ToolCapability object in ToolRegistry.tool_map.

[0042] If the ToolCapability object corresponding to the current candidate application's app_id does not exist in ToolRegistry.tool_map, it means the system has not registered a standardized tool interface for that candidate application, and tool_capability_check() continues to check the next candidate application. If a corresponding ToolCapability object is found, tool_capability_check() continues to check whether each input file type in input_mime_types belongs to ToolCapability.supported_mime_types. If there are mismatched input file types, it continues to check the next candidate application. If all input file types match, it continues to traverse required_capabilities and checks whether each capability exists in ToolCapability.capability_list. When the ToolCapability.capability_list corresponding to a candidate application completely covers required_capabilities, tool_capability_check() returns TOOL_AVAILABLE, agent-workspaced records the candidate application as selected_tool_app, sets the task status to TASK_TOOL_EXECUTION, and assigns the task to the corresponding standardized interface for execution. During the execution of the standardized interface, agent-workspaced passes the user-authorized input file and authorized output directory as tool call parameters, and records audit events such as TOOL_CALLED, TOOL_FILE_ACCESS, and TOOL_RESULT_CREATED through Audit.AppendEvent. After the result file is generated, agent-workspaced continues to call the Result.Finalize and Handoff interfaces to complete the generation of result proof, user confirmation, and result feedback, so that the standardized interface execution branch also enters the unified audit binding and result delivery process.

[0043] If none of the candidate applications can cover required_capabilities, tool_capability_check() writes the missing capabilities to the missing_capabilities field and returns TOOL_UNAVAILABLE. agent-workspaced sets the task status to TASK_GRAPHICAL_ISOLATION_REQUIRED and enters the isolated graphics execution flow. Subsequent steps mainly describe the implementation of entering the isolated graphics execution flow when TOOL_UNAVAILABLE is selected.

[0044] Step 4: Generate task-level policy configuration files and create independent Agent identities and task execution directories.

[0045] Once a task enters the isolated graphical execution flow, for example, agent-workspaced creates the task root directory / var / lib / agent-workspace / task_id to ensure that subsequent task-level policy configuration files can be written to this task directory. After the task root directory is created, agent-workspaced calls policy_generate(task_id) to generate the task-level policy configuration file AgentPolicy based on AgentTask, user authorization information, missing_capabilities, and default-policy.json. AgentPolicy contains the fields task_id, agent_identity, allowed_files, allowed_apps, output_dir, network_enable, clipboard_enable, audit_enable, handoff_policy, and missing_capabilities. The `allowed_files` directive comes from user-authorized input files, `allowed_apps` comes from task parsing results and user-authorized applications, `output_dir` points to the isolated output directory of the current task, `network_enable` is set to `false`, `clipboard_enable` is set to `false`, `audit_enable` is set to `true`, `handoff_policy` is set to `REQUIRE_USER_CONFIRM`, and `missing_capabilities` records capabilities that cannot be covered by the standardized tool interface and need to be completed through the graphical interface. `policy_generate()` serializes the `AgentPolicy` and writes it to ` / var / lib / agent-workspace / task_id / policy.json`. This `policy.json` is subsequently read by the `FileGateway`, `AppBroker`, `UIAdapter`, `Result`, and `Handoff` modules as the basis for all allow or deny decisions. After the policy configuration file is generated, `agent-workspaced` calls `workspace_create(task_id)`. `workspace_create()` creates a low-privilege Agent identity `agent-task-task_id` based on `task_id`. This agent identity is distinct from the user's main identity, does not belong to the user's main account, and does not have permission to access the user's complete home directory.Subsequently, `workspace_create()` creates the subdirectories `input`, `work`, `output`, `audit`, `home`, and `run` in the task's root directory. The `input` directory stores authorized input file mappings, the `work` directory stores intermediate task files, the `output` directory stores result files, the `audit` directory stores audit events and result verification information, the `home` directory serves as the Agent process's HOME directory, and the `run` directory serves as the runtime directory for the independent graphical session. After directory creation, `workspace_create()` sets the task directory owner to `agent-task-task_id`. The `input`, `work`, `output`, `home`, and `run` directories allow Agent identity access according to policies. The `audit` directory is written by `agent-workspaced`, and the Agent process cannot modify audit events already written.

[0046] Step 5: The FileGateway module establishes authorized file mappings and enforces access rules.

[0047] After the task directory is created, agent-workspaced internally calls the org.openkylin.AgentWorkspace1.FileGateway.PrepareMounts interface, for example. The input parameters for PrepareMounts are task_id and allowed_files. The FileGateway interface reads the allowed_files field from policy.json and establishes a mapping for each authorized file.

[0048] For each authorization file, the FileGateway module mounts the file from the user's actual path to the corresponding mapped path in the ` / var / lib / agent-workspace / task_id / input / ` directory using read-only bindmount. After mounting, the Agent process can only access the authorization file through the mapped path in the `input` directory, and cannot directly access the user's original path. The FileGateway module also generates a `file_map.json` file, which contains the original path, mapped path, access permissions, file size, and initial hash of the input file.

[0049] Subsequently, the FileGateway module generates an AppArmor profile and mount namespace rules for the task. These rules only allow the agent to access the ` / var / lib / agent-workspace / task_id / input, work, output, home, and run` directories, and deny access to unauthorized paths in the user's home directory, browser configuration directories, key directories, and other paths not authorized by `policy.json`. After the authorization file mapping is complete, FileGateway registers a fanotify listener for the current task's input, work, and output directories to capture open, read, write, and close events of the agent process on files within the authorization file or task directory. When the agent process accesses a file within the authorized scope, FileGateway calls `Audit.AppendEvent` to write a `FILE_ALLOWED` event based on the PID, path, and access type returned by `fanotify`. When the agent process accesses an unauthorized path, AppArmor denies access based on the profile corresponding to the current task, the kernel audit subsystem generates a denial record, and FileGateway calls `Audit.AppendEvent` to write a `FILE_DENIED` event based on the PID, path, and access type in the denial record. In this way, both allow and deny access are transformed into structured audit events and written into the audit event chain.

[0050] Step 6: Start an independent Wayland graphical session, create a task-level UI Socket, and control the launch of authorized applications through the AppBroker module.

[0051] After the authorization file mapping is complete, for example, agent-workspaced calls wayland_session_start(task_id). This function starts a standalone Wayland compositor with the identity agent-task-task_id and sets XDG_RUNTIME_DIR to / var / lib / agent-workspace / task_id / run. wayland_session_start() generates a standalone display socket WAYLAND_DISPLAY=agent-wayland-task_id for the current task and writes this value to / var / lib / agent-workspace / task_id / context.json. Subsequent graphical applications started by the AppBroker module use this WAYLAND_DISPLAY. The user's main desktop continues to use the original graphical session; the Agent graphical session does not share the window tree, input events, or clipboard with the user's main desktop.

[0052] After the independent Wayland graphical session starts, agent-workspaced creates a task-level Unix DomainSocket: / run / agent-workspace / task_id / ui.sock. This socket is listened to by the graphical operation adaptation module and provides two types of messages: GetUIState and InjectInputEvent. GetUIState is used to obtain the window, control, and screenshot states of the independent graphical session; InjectInputEvent is used to inject mouse, keyboard, and window control events into the independent graphical session. The socket owner is set to agent-task-task_id, and only the AI ​​agent corresponding to the current task can connect. After wayland_session_start() completes, agent-workspaced calls Audit.AppendEvent to write the SESSION_STARTED audit event. When the AI ​​agent executes a task, it needs to launch the graphical application and must call the D-Bus interface org.openkylin.AgentWorkspace1.AppBroker.LaunchApp. The input parameters of LaunchApp include task_id, app_id, and argv.

[0053] After receiving a LaunchApp request, the AppBroker module reads ` / var / lib / agent-workspace / task_id / policy.json` and checks if the `app_id` exists in `allowed_apps`. If the `app_id` is not in `allowed_apps`, the AppBroker module refuses to launch, returns `APP_DENIED`, and calls `Audit.AppendEvent` to write the `APP_DENIED` audit event. If the `app_id` exists in `allowed_apps`, the AppBroker module does not directly trust the executable path passed in by the AI ​​agent. Instead, it queries the system application registry or desktop file index based on the `app_id` to obtain the canonicalized executable file path and launch parameter template corresponding to that `app_id`. The AppBroker module only allows the launch of programs corresponding to this canonicalized path and prevents the AI ​​agent from overwriting executable file paths via `argv`, injecting shell commands, or launching unauthorized binaries.

[0054] After successful verification, the AppBroker module constructs a controlled startup environment: UID is set to agent-task-task_id, HOME is set to / var / lib / agent-workspace / task_id / home, working directory is set to / var / lib / agent-workspace / task_id / work, WAYLAND_DISPLAY is set to agent-wayland-task_id, and XDG_RUNTIME_DIR is set to / var / lib / agent-workspace / task_id / run. Subsequently, the AppBroker module starts the authorized application with the agent-task-task_id identity and writes the process PID to / var / lib / agent-workspace / task_id / app_pids.json. After startup, the AppBroker module calls Audit.AppendEvent to write the APP_LAUNCHED audit event. If the AI ​​agent requests to start an unauthorized application or passes invalid startup parameters, the AppBroker module refuses to start the application and records the APP_DENIED event.

[0055] Step 7: The graphics operation adaptation module performs graphics operations, and the Audit module generates a hash chain of audit events.

[0056] After the authorized application starts, for example, the AI ​​agent connects to ` / run / agent-workspace / task_id / ui.sock` and sends a `GetUIState` message. The `GetUIState` message contains the `task_id` and the target application window identifier. Upon receiving the request, the graphical operation adaptation module reads the window title, window hierarchy, focused window, control area, screen image, and application response state from the Wayland compositor control interface corresponding to `agent-wayland-task_id`, and returns the interface state to the AI ​​agent. The AI ​​agent generates graphical operation instructions based on the interface state and sends an `InjectInputEvent` message through the same `ui.sock`. The `InjectInputEvent` message contains `task_id`, `event_type`, `target_window`, `position`, `keycode`, and `text` fields. Upon receiving this message, the graphical operation adaptation module first reads `app_pids.json` and the current Wayland window ownership information to determine if `target_window` belongs to the authorized application window launched by the current task. If `target_window` does not belong to the authorized application window, the graphical operation adaptation module rejects the injected event and calls `Audit.AppendEvent` to write the `UI_DENIED` audit event. If `target_window` belongs to an authorized application window, the graphical operation adaptation module injects mouse events, keyboard events, or window control events into the `agent-wayland-task_id` independent graphical session and calls `Audit.AppendEvent` to write the `UI_ACTION` audit event. Through this process, the AI ​​agent's graphical operations only affect the independent graphical session and not the user's main desktop.

[0057] In this solution, all critical modules write audit logs through the `org.openkylin.AgentWorkspace1.Audit.AppendEvent` interface. The input parameters for `AppendEvent` include `task_id`, `event_type`, `object`, `result`, and `detail`. Upon receiving an `AppendEvent` call, the Audit module reads the hash `prev_hash` of the previous audit event for the current task and constructs a new `AuditEvent`. The `AuditEvent` contains the `task_id`, `event_seq`, `event_type`, `event_time`, `agent_uid`, `object`, `result`, `detail`, and `prev_hash`. Subsequently, the Audit module calculates `curr_hash = SHA256(prev_hash||event_seq||event_type||event_time||agent_uid||object||result||detail)` and appends the `AuditEvent` to ` / var / lib / agent-workspace / task_id / audit / events.log`. After the write operation is complete, the Audit module updates curr_hash to the current task's audit_final_hash and writes it to / var / lib / agent-workspace / task_id / audit / audit_state.json. In this way, events such as FILE_ALLOWED, FILE_DENIED, APP_LAUNCHED, APP_DENIED, SESSION_STARTED, UI_ACTION, and UI_DENIED form an audit event chain in chronological order.

[0058] Step 8: The Result module generates result verification information, and the Handoff module performs user confirmation, result feedback, and task cleanup.

[0059] After the AI ​​agent completes its task, the result file is written to the ` / var / lib / agent-workspace / task_id / output` directory, for example. `agent-workspaced` monitors the `output` directory. When a result file matching the `output_type` is detected, the D-Bus interface `org.openkylin.AgentWorkspace1.Result.Finalize` is called. The input parameters for `Finalize` are `task_id` and `result_path`. Upon receiving the `Finalize` call, the `Result` module first performs a path normalization check on `result_path`, rejecting symbolic links, hard links, or paths pointing outside the task directory, and confirming that the actual path of the result file is located in the ` / var / lib / agent-workspace / task_id / output` directory. After the path check passes, the `Result` module reads the result file header and extension to confirm that the file type matches the task output type. It then calculates the SHA256 hash of the result file and reads the initial hash of the input file from `file_map.json`. Next, the Result module reads events.log, audit_state.json, app_pids.json, and policy.json, and summarizes the list of authorized applications, the list of actually launched applications, file access records, graphical operation records, exception denial records, and the final hash of the audit event chain.

[0060] The Result module writes the above information to ` / var / lib / agent-workspace / task_id / audit / result_manifest.json`. This `result_manifest.json` is in a pending confirmation state, with the `user_confirmed` field set to false. `result_manifest.json` contains `task_id`, `agent_uid`, original input file path, mapped input file path, input file hash, list of authorized applications, list of actually launched applications, file access event summary, graph operation event summary, exception rejection event summary, final hash of the audit event chain, result file path, and result file hash. The Result module then calculates the hash of `result_manifest.json` and generates a pending confirmation version, `result.proof`. `result.proof` contains the result file hash, result proof information hash, and the final hash of the current audit event chain. At this point, the result file is initially bound to the task policy, execution process, exception rejection events, and the current audit chain, but it does not yet contain user confirmation feedback events.

[0061] After the result verification information is generated, the user's main desktop agent frontend calls the `org.openkylin.AgentWorkspace1.Handoff.Prepare` interface to obtain the result file path, `result_manifest.json`, and `result.proof`, and displays them to the user. At this time, the result file is still located in the isolated output directory and has not yet been written to the user's home directory. When the user confirms receipt of the result, the user's main desktop agent frontend calls `org.openkylin.AgentWorkspace1.Handoff.Commit`. The input parameters for `Commit` include `task_id` and `target_dir`. After receiving the `Commit` call, the Handoff module first recalculates the hash of the result file in the isolated output directory and compares it with the `result_hash` recorded in `result_manifest.json`. After the hash comparison passes, the Handoff module copies the result file to a temporary file in the user's target directory and calculates the hash of the temporary file to confirm that it matches the `result_hash`. After confirming that the copied temporary file has not changed, the Handoff module calls `Audit.AppendEvent` to write the `HANDOFF_CONFIRMED` audit event, causing the user's confirmation action to enter the audit event chain.

[0062] Subsequently, the Handoff module reads the updated audit_final_hash, sets the user_confirmed field in result_manifest.json to true, and writes the target_dir, postback time, and updated audit_final_hash into result_manifest.json. Then, the Handoff module recalculates the hash of result_manifest.json and regenerates the final version of result.proof, ensuring that the final result.proof simultaneously binds the result file hash, the updated result proof information hash, and the final hash of the audit event chain containing the HANDOFF_CONFIRMED event.

[0063] After the final version of `result.proof` is generated, the Handoff module writes the updated `result_manifest.json` and the final version of `result.proof` to the user-specified directory, and renames the temporary result file in the target directory to the official result file. Upon delivery, the Handoff module updates the task status to `TASK_DONE`. When the user refuses to receive the result, the user's main desktop agent frontend calls `org.openkylin.AgentWorkspace1.Handoff.Reject`. The Handoff module calls `Audit.AppendEvent` to write the `HANDOFF_REJECTED` audit event and updates the task status to `TASK_REJECTED`. When the user cancels the task, the `Manager.CancelTask` interface updates the task status to `TASK_CANCELLED` and writes the `TASK_CANCELLED` audit event.

[0064] Once a task enters the TASK_DONE, TASK_REJECTED, or TASK_CANCELLED state, agent-workspaced calls workspace_cleanup(task_id) to perform task cleanup. workspace_cleanup(task_id) terminates the application process and independent Wayland graphical session started by the task, closes the task-level ui.sock, removes the binding mount of the authorization file, disables the AppArmorprofile and mount namespace rules corresponding to the task, releases or disables the agent-task-task_id identity, and archives the audit event chain, result_manifest.json, and result.proof in the audit directory according to the retention policy. Through this cleanup process, the authorization file mapping, graphical session, and Agent execution permissions are revoked after the task ends, preventing residual environments from continuing to access user resources after the task is completed.

[0065] Through the above process, this solution creates task management, application startup, file authorization, auditing, result generation, and result feedback interfaces during the service initialization phase, and constructs a tool capability index. During each task creation phase, it generates a task-level policy configuration file, an independent Agent identity, an isolated directory, an independent Wayland graphical session, and a task-level ui.sock. The AI ​​agent's application startup, file access, interface state acquisition, input event injection, result generation, and result feedback are all executed under control through these interfaces. Each interface call reads the policy configuration file corresponding to the task_id and writes it into the audit event chain after allowing or denying the request. Thus, the AI ​​agent's task authorization, execution process, exception interception, result files, and user confirmation actions are bound together into a verifiable chain, enabling the AI ​​agent's graphical task results to be traceable, verifiable, and auditable.

[0066] Example 3: This embodiment is a specific application example of the method flow described in Embodiment 2, and is a reliable proof example of the AIAgent table sorting results based on the openKylin desktop system.

[0067] This embodiment is implemented based on the openKylin desktop operating system, which uses the Wayland graphics architecture and deploys the system-level persistent service agent-workspaced. Assume the user is currently creating a PowerPoint presentation on their main desktop and submits a task to the AIAgent: "Please organize the source.xlsx table in the Documents directory, remove duplicates, adjust the format, and export it as a PDF." When initiating the task, the user authorizes the AI ​​Agent to access / home / user / Documents / source.xlsx and allows the AI ​​Agent to complete the task using LibreOffice Calc or WPS Spreadsheet. By default, the system prohibits the AIAgent from accessing the network, the user's full home directory, launching terminals, browsers, system settings, and password managers. The system also requires the result file to be written to an isolated output directory first, and then sent back to the user's main desktop directory only after user confirmation.

[0068] The specific execution steps are as follows: 1. The Agent frontend on the user's main desktop calls the `org.openkylin.AgentWorkspace1.Manager.SubmitTask` interface to send a task request to `agent-workspaced`. The `SubmitTask` request includes the user identifier, task description, authorized input file ` / home / user / Documents / source.xlsx`, candidate applications `libreoffice-calc` and `wps-office`, and output type `PDF`. After receiving the request, `agent-workspaced` calls `task_create()` to create a task object and generates a task identifier `task-001`.

[0069] 2. agent-workspaced calls task_analyze() to parse the task content, determines that the input file for this task is / home / user / Documents / source.xlsx, the input file type is xlsx, the candidate applications are libreoffice-calc and wps-office, the output type is PDF, and the required capabilities include opening the file, modifying table content, deleting duplicates, adjusting table format, and exporting PDF.

[0070] 3. The agent-workspaced calls tool_capability_check(task-001) to check the standardized tool interface capabilities corresponding to the candidate application. The system sequentially checks whether libreoffice-calc and wps-office declare in the tool capability index the ability to process xlsx files and perform duplicate removal, format adjustment, and PDF export. In this embodiment, the standardized tool interface does not fully cover the duplicate removal and format adjustment capabilities, so tool_capability_check() returns TOOL_UNAVAILABLE. The agent-workspaced sets the task status to TASK_GRAPHICAL_ISOLATION_REQUIRED and enters the isolated graphics execution flow.

[0071] 4. agent-workspaced creates the task root directory / var / lib / agent-workspace / task-001 and generates the task-level policy configuration file / var / lib / agent-workspace / task-001 / policy.json. This policy configuration file records: task identifier as task-001; agent identity as agent-task-task-001; authorized input file as / home / user / Documents / source.xlsx; authorized applications as libreoffice-calc and wps-office; output directory as / var / lib / agent-workspace / task-001 / output; network access and clipboard sharing are both prohibited; the result feedback policy requires user confirmation; capabilities that need to be completed through the graphical interface include deleting duplicates and adjusting formatting.

[0072] 5. The agent-workspaced creates a low-privilege Agent identity, agent-task-task-001, and creates the subdirectories input, work, output, audit, home, and run under / var / lib / agent-workspace / task-001. Input stores the mapping of authorized input files, work stores intermediate task files, output stores the result files, audit stores the audit event chain and result verification information, home serves as the Agent process's HOME directory, and run serves as the runtime directory for independent graphical sessions.

[0073] 6. FileGateway maps ` / home / user / Documents / source.xlsx` to ` / var / lib / agent-workspace / task-001 / input / source.xlsx` using read-only bindmount based on `policy.json`, and generates `file_map.json`, recording the original path, mapped path, access permissions, file size, and initial hash of the input file. Subsequently, FileGateway configures AppArmor profile and mount namespace rules for `task-001`, ensuring that `agent-task-task-001` can only access the `input`, `work`, `output`, `home`, and `run` directories within the current task directory, and cannot access unauthorized paths such as the user's full home directory, browser configuration directory, and key directory.

[0074] 7. FileGateway registers file access listeners for the input, work, and output directories of task-001. When the AI ​​agent accesses input / source.xlsx, FileGateway logs a FILE_ALLOWED audit event; when the AI ​​agent attempts to access / home / user / .ssh, / home / user / .config, or other unauthorized directories, AppArmor denies access, and FileGateway writes a FILE_DENIED audit event based on the denial record.

[0075] 8. `agent-workspaced` starts the independent Wayland graphical session corresponding to `task-001` and sets `WAYLAND_DISPLAY=agent-wayland-task-001`. This graphical session does not share the window tree, input events, or clipboard with the user's main desktop. `agent-workspaced` also creates ` / run / agent-workspace / task-001 / ui.sock` for the AI ​​agent to obtain the interface state and inject input events. After the graphical session starts, the system writes a `SESSION_STARTED` audit event.

[0076] 9. When the AI ​​agent requests to launch LibreOfficeCalc, it calls the `org.openkylin.AgentWorkspace1.AppBroker.LaunchApp` interface and submits `task_id=task-001` and `app_id=libreoffice-calc`. The AppBroker reads `policy.json`, confirms that `libreoffice-calc` is an authorized application, obtains its canonical executable path based on the system application registration information, and prevents the AI ​​agent from overwriting the executable path via `argv` or injecting shell commands. After successful verification, the AppBroker launches LibreOfficeCalc in the `agent-wayland-task-001` graphical session as `agent-task-task-001` and writes the `APP_LAUNCHED` audit event. If the AI ​​agent requests to launch an unauthorized application such as a terminal, browser, or system settings application, the AppBroker refuses to launch it and writes the `APP_DENIED` audit event.

[0077] 10. After LibreOfficeCalc starts, the AI ​​agent sends a GetUIState message via / run / agent-workspace / task-001 / ui.sock to obtain the window title, focused window, control area, screen image, and application response state in the independent graphical session. The AI ​​agent generates operation instructions based on the interface state and injects mouse, keyboard, or window control events into the independent graphical session via the InjectInputEvent message.

[0078] 11. In this embodiment, the AI ​​agent sequentially completes operations such as opening input / source.xlsx, selecting a data region, removing duplicates, adjusting table format, selecting export to PDF, and saving to output / result.pdf within an independent graphical session. The graphical operation adaptation module verifies whether the target window belongs to the authorized application window launched by task-001 before each injection event. If it belongs to the authorized window, injection is performed and a UI_ACTION audit event is written; if it does not belong to the authorized window, injection is rejected and a UI_DENIED audit event is written.

[0079] 12. During task execution, the Audit module writes events such as FILE_ALLOWED, FILE_DENIED, APP_LAUNCHED, APP_DENIED, SESSION_STARTED, UI_ACTION, and UI_DENIED to / var / lib / agent-workspace / task-001 / audit / events.log in chronological order. Each audit event includes the hash of the previous event and the hash of the current event. The hash of the current event is calculated from the hash of the previous event and the content of the current event. The Audit module also writes the final hash of the audit event chain of the current task to audit_state.json.

[0080] 13. After the AI ​​agent completes the table organization and exports the PDF, the result file is saved to / var / lib / agent-workspace / task-001 / output / result.pdf. The agent-workspaced detects this result file and calls Result.Finalize. The Result module first performs a path normalization check on result.pdf to confirm that it is not a symbolic link, hard link, or path pointing outside the task directory, and confirms that the actual path is located in the output directory of task-001. Subsequently, the Result module checks whether the file type is PDF, calculates the SHA256 hash of result.pdf, and reads the hash of the source.xlsx input file recorded in file_map.json.

[0081] 14. The Result module reads events.log, audit_state.json, app_pids.json, and policy.json, generating / var / lib / agent-workspace / task-001 / audit / result_manifest.json. This file records the task-001 task identifier, agent identity, original input file path, input file mapping path, input file hash, authorized application list, actual launched application list, file access event summary, graphical operation event summary, exception rejection event summary, final hash of the audit event chain, result file path, and result file hash. At this point, the user_confirmed field is false. The Result module then calculates the hash of result_manifest.json and generates a pending confirmation version, result.proof, which is used to bind the result file hash, result proof information hash, and the final hash of the current audit event chain.

[0082] 15. After the result verification information is generated, the user's main desktop agent frontend calls the Handoff.Prepare interface to obtain result.pdf, result_manifest.json, and result.proof, and displays them to the user. At this time, result.pdf is still located in the isolated output directory and has not yet been written to the user's home directory.

[0083] 16. After the user confirms receipt of the result, the user's main desktop Agent frontend calls the Handoff.Commit interface, specifying the return directory / home / user / Documents / AgentResults. The Handoff module recalculates the hash of result.pdf in the isolated output directory and compares it with the hash of the result file recorded in result_manifest.json. After the comparison is successful, the Handoff module copies result.pdf to a temporary file / home / user / Documents / AgentResults / result.pdf.tmp in the user's target directory, and recalculates the hash of this temporary file to confirm that the copying process has not changed the content of the result file.

[0084] 17. After the temporary file verification passes, the Handoff module writes the HANDOFF_CONFIRMED audit event, causing the user confirmation action to enter the audit event chain. Subsequently, the Handoff module reads the updated final hash of the audit event chain, updates the user_confirmed field in result_manifest.json to true, and writes the return directory, return time, and the updated final hash of the audit event chain. Afterward, the Handoff module recalculates the hash of result_manifest.json and regenerates the final version of result.proof, ensuring that the final result.proof simultaneously binds the result file hash, the updated result proof information hash, and the final hash of the audit event chain containing the user confirmation event.

[0085] 18. After the final version of result.proof is generated, the Handoff module writes the updated result_manifest.json and the final version of result.proof to the / home / user / Documents / AgentResults directory, and renames result.pdf.tmp to the official result file result.pdf. After delivery, agent-workspaced updates the task status to TASK_DONE. If the user refuses to receive the results, the Handoff module writes the HANDOFF_REJECTED audit event and updates the task status to TASK_REJECTED.

[0086] 19. After the task status enters TASK_DONE or TASK_REJECTED, agent-workspaced calls workspace_cleanup(task-001) to perform task cleanup. workspace_cleanup(task-001) terminates the LibreOfficeCalc process and independent Wayland graphical session started by task-001, closes / run / agent-workspace / task-001 / ui.sock, removes the bindmount mapping of source.xlsx, disables the AppArmor profile and mount namespace rules corresponding to task-001, releases or disables the agent-task-task-001 identity, and archives events.log, audit_state.json, result_manifest.json, and result.proof in the audit directory according to the retention policy.

[0087] As can be seen from the above embodiments, this solution does not simply record operation logs when the AI ​​Agent executes graphical software tasks. Instead, it binds task-level policy configuration files, authorized input files, authorized applications, independent graphical sessions, file access events, application startup events, graphical operation events, exception rejection events, result file hashes, user confirmation records, and result proof information. After receiving the PDF file generated by the AI ​​Agent, the user can trace the input source, execution process, unauthorized access interception, user confirmation status, and file integrity of the file based on result_manifest.json and result.proof, thereby improving the credibility and auditability of the AI ​​Agent's graphical operation results.

[0088] The above-described embodiments are merely preferred embodiments of the present invention and are only used to help understand the method and core ideas of this application. The scope of protection of the present invention is not limited to the above embodiments, and all technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. An AI intelligent agent desktop task execution system based on isolated graphics execution, characterized in that, It includes a task management module, a tool capability index module, a file authorization gateway module, an application startup agent module, a graphical operation adaptation module, an audit module, a result processing module, and a result feedback module. All modules together constitute the functional logic body of the system's resident process, agent-workspaced. agent-workspaced receives user requests and creates AI agent tasks through the task management module. It uses the capability index of local tools built by the tool capability index module to determine whether the AI ​​agent task can be completed through a standardized interface. If not, it switches to isolated graphical execution. An isolated environment and access control for the isolated graphical interface are constructed through the file authorization gateway module and the application launch agent module; the state of the isolated graphical interface is obtained and operation events are injected through the graphical operation adaptation module; the operation events are recorded through the audit module and a hash-based anti-tampering audit event chain is generated; the result processing module calculates the hash of the result file of the AI ​​agent task and generates result proof information; and the result feedback module displays the AI ​​agent task result to the user, allows the user to confirm the feedback result, or refuse to feedback the result.

2. The AI ​​intelligent agent desktop task execution system based on isolated graphics execution according to claim 1, characterized in that, The task management module generates a unique task identifier (task_id) for each AI agent task, serving as a unified index for the execution process in the isolated graphical interface.

3. The AI ​​intelligent agent desktop task execution system based on isolated graphics execution according to claim 1, characterized in that, The tool capability index module loads the system's local tool capability description file, parses the capability, and builds a capability index. Before loading, it first creates a ToolCapability structure and a ToolRegistry structure in memory. The ToolCapability structure is used to describe the capability information of a single tool, and the ToolRegistry structure is used to store the capability information of all loaded tools.

4. The AI ​​intelligent agent desktop task execution system based on isolated graphics execution according to claim 1, characterized in that, The file authorization gateway module controls the file access scope of the AI ​​agent and maps authorized files to isolated directories.

5. The AI ​​intelligent agent desktop task execution system based on isolated graphics execution according to claim 1, characterized in that, The control AI agent launches a graphical application authorized via a task-level policy configuration file, which is generated by agent-workspaced based on the AI ​​agent's task, user authorization, missing capabilities of local tools, and default policy template.

6. The method of using the AI ​​intelligent agent desktop task execution system based on isolated graphics execution as described in any one of claims 1-5, characterized in that, include: S1. The system resident process agent-workspaced connects to D-Bus and registers the D-Bus interfaces of the corresponding task management module, application startup agent module, file authorization gateway module, audit module, result processing module, and result return module. S2 and agent-workspaced load local tool capability description files through the tool capability index module, parse the files, and build an in-memory capability index. S3. agent-workspaced receives the user's AI agent task request through the task management module, creates the AI ​​agent task and matches it according to the capability index. If the match is successful, the task is completed through the standardized interface; otherwise, it proceeds to the isolated graphical execution process in step S4. S4 and agent-workspaced create root and isolation directories for AI agent tasks and dynamically generate task-level policy configuration files that configure authorization files, authorized applications, output directories, and disabled items. S5. Map authorized files to isolated directories through the file authorization gateway module and establish mandatory access control rules; S6. Start an isolated Wayland standalone graphical session and launch the application through the application launch agent module; S7. Obtain the interface state of the Wayland independent graphical session through the graphical operation adaptation module and send it to the AI ​​agent, and inject the operation events of the AI ​​agent into the interface of the Wayland independent graphical session; record the operation events of the AI ​​agent through the audit module and generate a hash chain-based anti-tampering audit event chain; S8. After the task is completed, the result processing module verifies the result file path and type, calculates the hash, and generates result proof information; the result feedback module displays the result to the user and terminates the AI ​​agent task.

7. The method of use according to claim 6, characterized in that, After the D-Bus interface is registered in step S1, agent-workspaced configures access control policies for each interface, creates a service working directory and a runtime directory, and loads the default policy template used to generate task-level policy configuration files.

8. The method of use according to claim 6, characterized in that, In step S2, before loading the tool capability description file, agent-workspaced first creates a ToolCapability structure and a ToolRegistry structure in memory. The ToolCapability structure is used to describe the capability information of a single tool, and the ToolRegistry structure is used to store the capability information of all loaded tools.

9. The method of use according to claim 6, characterized in that, In step S3, the task management module generates a unique task identifier, task_id, which serves as a unified index for subsequent steps.

10. The method of use according to claim 6, characterized in that, The dynamic generation of the task-level policy configuration file in step S4 includes: A task-level policy configuration file is generated based on the AI ​​agent's task, user authorization information, missing information about local tool capabilities, and the default policy template.

Images