Remote control method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a remote control method and device, electronic equipment, a storage medium and a program product, relates to the technical field of control, and is used for improving the efficiency of terminal control. The method comprises the steps that a target server receives a task request message from an AI agent, the task request message is used for requesting a target terminal to use a target tool to execute a target operation, and the task request message indicates a tool identifier corresponding to the target tool and an operation parameter corresponding to the target operation; according to the tool identifier and the operation parameter, a target task is generated, and the target task is a task for controlling the target terminal to execute the target operation based on the target tool; and putting the target task into a target task queue corresponding to the target terminal, and pushing the task in the target task queue to the client. The method is used in a scene of remotely controlling the personal terminal.

Description

Technical Field

[0001] This application relates to the field of control technology, and in particular to remote control methods, devices, electronic devices, storage media, and program products. Background Technology

[0002] Currently, users can remotely control terminals using artificial intelligence (AI) agents to enable the terminals to perform various office automation tasks. Specifically, a Model Context Protocol Server (MCP Server) can be started on the terminal first, and then the AI ​​agent can be configured to connect to the MCP Server. In this way, users can issue control commands to the AI ​​agent, which can then perform multiple signaling interactions with the terminal based on these control commands to invoke the corresponding local tools on the MCP Server to control the terminal.

[0003] However, since the AI ​​agent needs to perform multiple signaling interactions with the terminal before calling the corresponding local tool on the MCP Server to control the terminal, the waiting time before controlling the terminal may be long, resulting in a longer terminal control time and thus affecting the efficiency of terminal control. Summary of the Invention

[0004] This application provides a remote control method, device, electronic device, storage medium, and program product to improve the efficiency of terminal control.

[0005] In a first aspect, this application provides a remote control method applied to a target server, which is communicatively connected to an AI agent and to a client running on a target terminal. The remote control method includes: the target server receiving a task request message from the AI ​​agent, the task request message being used to request the target terminal to perform a target operation using a target tool, the task request message indicating a tool identifier corresponding to the target tool and operation parameters corresponding to the target operation; generating a target task based on the tool identifier and operation parameters, the target task being a task to control the target terminal to perform a target operation based on the target tool; and placing the target task into a target task queue corresponding to the target terminal and pushing the task in the target task queue to the client.

[0006] The technical solution provided in this application brings at least the following beneficial effects: When the target server receives a task request message from the AI ​​agent requesting the target terminal to use the target tool to perform a target operation, it can generate a target task to control the target terminal to perform the target operation based on the tool identifier and operation parameters in the task request message. In this way, the target server can put the target task into the target task queue corresponding to the target terminal, so as to directly push the task in the target task queue to the client running on the target terminal. That is, it can directly push the target task to the client without having to perform multiple signaling interactions with the target terminal. Therefore, it can reduce the waiting time before controlling the target terminal, thereby reducing the time spent controlling the target terminal and improving the efficiency of terminal control.

[0007] In one possible implementation, before generating the target task based on the tool identifier and tool parameters, the remote control method further includes: the target server obtaining the client identifier corresponding to the client; the generation of the target task based on the tool identifier and tool parameters includes: the target server generating a target task identifier based on the client identifier, tool identifier, and tool parameters corresponding to the client, which is used to indicate the target task; and encapsulating the client identifier, tool identifier, tool parameters, and target task identifier to obtain the target task.

[0008] Thus, it can be seen that since the target server can obtain the client identifier corresponding to the client and generate a target task identifier based on the client identifier, tool identifier, and tool parameters, after encapsulation, the target task can be associated with the client identifier, target task identifier, tool identifier, and tool parameters. Therefore, on the one hand, the target server can accurately add the target task to the target task queue corresponding to the target terminal, so as to accurately push the target task to the target terminal. On the other hand, it can enable the target terminal to know the association between the target task and the target task identifier. In subsequent steps, the target terminal can associate the execution result with the target task identifier and report it to the target server, so that the target server can accurately know the task corresponding to the execution result, which is convenient for the target server to accurately know the execution result of the target task.

[0009] Another possible implementation is that the task request message includes a call request for a target proxy function, which is a proxy function of the tool function of the target tool; before generating the target task based on the tool identifier and tool parameters, the remote control method further includes: the target server determining the tool identifier and operation parameters based on the call request and the target proxy function.

[0010] Thus, since the target server can determine the tool identifier and operation parameters based on the call request and the target proxy function, it is not necessary to deploy specific tool functions on the target server. Therefore, code redundancy can be reduced and storage space on the target server can be saved.

[0011] Another possible implementation, before determining the tool identifier and operation parameters based on the call request and the target proxy function, further includes: the target server receiving a tool registry from the client, the tool registry including tool functions of at least one tool in the target terminal, the at least one tool including the target tool; and calling the make_proxy function for each tool's tool function to generate a proxy function for each tool's tool function, obtaining at least one proxy function, the at least one proxy function including the target proxy function.

[0012] Thus, since the target server can receive the tool registry from the client and call the make_proxy function for the tool function of each tool in the tool registry to generate the proxy function of the tool function of each tool, it is not necessary to deploy the specific tool function of any tool in the target terminal on the target server. Therefore, code redundancy can be reduced and storage space on the target server can be saved.

[0013] Another possible implementation is that, after pushing the task objects in the target task queue to the client as described above, the remote control method further includes: the target server sending a heartbeat packet to the client when the target task queue is empty for a period of time greater than or equal to a time threshold.

[0014] Therefore, since the target server can send a heartbeat packet to the client when the target task queue is empty for a period of time greater than or equal to a time threshold (i.e., when no task has been pushed to the client for a relatively long time), it can avoid the situation where the target server or the target terminal disconnects due to the long period of no task being pushed to the client. This avoids the need to re-establish the connection between the two before the target server pushes a task to the client again, thus improving the efficiency and reliability of task push.

[0015] Another possible implementation is that, after pushing the task objects in the target task queue to the client as described above, the remote control method further includes: the target server receiving the target task execution result from the client, which is the execution result of the target task; and sending the target task execution result to the AI ​​agent.

[0016] Thus, it can be seen that when the target server receives the execution result of the target task, it can send the execution result to the AI ​​agent so that the AI ​​agent can know the execution result. In this way, the AI ​​agent can provide timely feedback to the user based on the execution result, so that the user can perform the next operation. Therefore, the convenience of terminal operation can be improved.

[0017] Another possible implementation is that the target task execution result includes a target task identifier, which is used to indicate the target task; before sending the target task execution result to the AI ​​agent, the remote control method further includes: the target server determines the target Future object associated with the target task identifier from at least one Future object based on the target task identifier; and stores the target task execution result in the target Future object.

[0018] Thus, it can be seen that since the target server can determine the target Future object associated with the target task identifier from at least one Future object based on the target task identifier, and store the target task execution result in the target Future object, this facilitates the target server to implement asynchronous task processing. Therefore, the target server can accurately obtain the target task execution result.

[0019] Another possible implementation is that, before determining the target Future object associated with the target task identifier from at least one Future object based on the target task identifier, the remote control method further includes: the target server creating the target Future object and establishing the association between the target Future object and the target task identifier.

[0020] Thus, since the target server can create a target Future object and establish an association between the target Future object and the target task identifier, it can facilitate the target server to implement asynchronous task processing. Therefore, the target server can accurately obtain the execution result of the target task.

[0021] Another possible implementation is that, after storing the target task execution result in the target Future object, the remote control method further includes: the target server deleting the target Future object from at least one Future object.

[0022] Thus, it can be seen that since the target server can delete the target Future object after sending the target task execution result to the AI ​​agent, that is, after the target task is completed, the target server's storage space can be saved.

[0023] Secondly, this application provides a remote control method applied to a target terminal, which runs a client and communicates with a target server. The remote control method includes: the target terminal receiving a target task from the target server through the client, the target task being a task to control the target terminal to perform a target operation based on a target tool; and performing the target operation based on the target tool according to the target task.

[0024] The technical solution provided in this application brings at least the following beneficial effects: Since the target terminal can directly receive the target task from the target server through the client to control the target terminal to perform the target operation based on the target tool, without having to perform multiple signaling interactions with the target server, the target terminal can quickly perform the target operation based on the target tool according to the target task. Therefore, the waiting time before controlling the target terminal can be reduced, thereby reducing the time spent controlling the target terminal and improving the efficiency of terminal control.

[0025] One possible implementation, wherein the above-mentioned execution of a target operation based on a target tool according to a target task, includes: the target terminal determining the tool identifier corresponding to the target tool and the operation parameters corresponding to the target operation according to the target task; and determining the tool function of the target tool from the tool registry according to the tool identifier, the tool registry including the tool function of at least one tool in the target terminal; and calling the tool function of the target tool using the operation parameters to execute the target operation using the target tool.

[0026] Thus, since the target terminal can accurately determine the tool function of the target tool from the tool registry based on the tool identifier, and accurately call the tool function of the target tool using the operation parameters, the target terminal can accurately execute the target operation using the target tool, thereby improving the accuracy of terminal operation.

[0027] Another possible implementation, before determining the tool identifier corresponding to the target tool and the operation parameters corresponding to the target operation based on the target task, includes the following steps in the remote control method: the target terminal registers at least one tool in the tool registry; and uses a decorator to mark the tool functions in the tool registry, and sends the marked tool registry to the target server; wherein the tool registry is used to create proxy functions for the tool functions.

[0028] Thus, since the target terminal can register at least one tool in the tool registry, mark the tool functions in the tool registry with decorators, and send the marked tool registry to the target server, the target server can create proxy functions for the tool functions of each tool. This eliminates the need to deploy the specific tool functions on the target server, thereby reducing code redundancy and saving storage space on the target server.

[0029] Another possible implementation is that, after performing the target operation based on the target tool according to the target task, the remote control method further includes: the target terminal sending the target task execution result to the target server through the client, and the target task execution result is the result of the target task execution.

[0030] Thus, since the target terminal can send the target task execution result to the target server through the client, the target server can know the target task execution result and promptly feed it back to the AI ​​agent. Therefore, the AI ​​agent can promptly provide feedback to the user based on the target task execution result, so that the user can perform the next operation. This improves the convenience of terminal operation.

[0031] Another possible implementation is that the target task execution result includes a target task identifier, which is used to indicate the target task; before sending the target task execution result to the target server through the client, the remote control method further includes: the target terminal determining the target task identifier according to the target task.

[0032] Thus, since the target terminal can determine the target task identifier based on the target task, in subsequent steps, the target terminal can associate the execution result with the target task identifier and report it to the target server, so that the target server can accurately know the task corresponding to the execution result. Therefore, it is convenient for the target server to accurately know the execution result of the target task.

[0033] Thirdly, this application provides a remote control device applied to a target server. The target server is communicatively connected to an AI agent and also communicatively connected to a client running on a target terminal. The remote control device includes: a receiving module for receiving a task request message from the AI ​​agent, the task request message requesting the target terminal to perform a target operation using a target tool, the task request message indicating a tool identifier corresponding to the target tool and operation parameters corresponding to the target operation; a generating module for generating a target task based on the tool identifier and operation parameters, the target task being a task that controls the target terminal to perform a target operation based on the target tool; an adding module for adding the target task generated by the generating module to a target task queue corresponding to the target terminal; and a sending module for pushing the task objects in the target task queue added by the adding module to the client.

[0034] Fourthly, this application provides a remote control device applied to a target terminal. The target terminal runs a client that communicates with a target server. The remote control device includes: a receiving module for receiving a target task from the target server via the client, the target task being a task to control the target terminal to perform a target operation based on a target tool; and an execution module for performing the target operation based on the target tool according to the target task received by the receiving module.

[0035] Fifthly, this application provides an electronic device comprising: a processor and a memory; the memory storing processor-executable instructions; when the processor is configured to execute the instructions, causing the electronic device to implement the method of the first aspect above, or to implement the method of the second aspect above.

[0036] In a sixth aspect, this application provides a computer-readable storage medium comprising: computer software instructions; when the computer software instructions are executed in an electronic device, they cause the electronic device to implement the method of the first aspect above, or to implement the method of the second aspect above.

[0037] In a seventh aspect, this application provides a computer program product comprising a computer program; when the computer program is run in an electronic device, it causes the electronic device to implement the method of the first aspect or the method of the second aspect described above.

[0038] The beneficial effects of the second to seventh aspects mentioned above are described in the corresponding description of the first aspect and will not be repeated here. Attached Figure Description

[0039] Figure 1 This is a schematic diagram illustrating the application environment of a remote control method provided in an embodiment of this application; Figure 2A flowchart illustrating a remote control method provided in an embodiment of this application; Figure 3 A flowchart illustrating another remote control method provided in an embodiment of this application; Figure 4 A flowchart illustrating yet another remote control method provided in an embodiment of this application; Figure 5 A flowchart illustrating yet another remote control method provided in an embodiment of this application; Figure 6 A flowchart illustrating yet another remote control method provided in an embodiment of this application; Figure 7 A flowchart illustrating yet another remote control method provided in an embodiment of this application; Figure 8 A flowchart illustrating yet another remote control method provided in an embodiment of this application; Figure 9 A flowchart illustrating yet another remote control method provided in an embodiment of this application; Figure 10 A flowchart illustrating yet another remote control method provided in an embodiment of this application; Figure 11 A flowchart illustrating yet another remote control method provided in an embodiment of this application; Figure 12 A flowchart illustrating yet another remote control method provided in an embodiment of this application; Figure 13 A flowchart illustrating yet another remote control method provided in an embodiment of this application; Figure 14 A flowchart illustrating yet another remote control method provided in an embodiment of this application; Figure 15 A flowchart illustrating yet another remote control method provided in an embodiment of this application; Figure 16 A flowchart illustrating yet another remote control method provided in an embodiment of this application; Figure 17 A schematic diagram of the composition of a remote control device provided in this application; Figure 18 A schematic diagram of another remote control device provided in this application; Figure 19 This is a schematic diagram of the composition of an electronic device provided in this application. Detailed Implementation

[0040] The following is a detailed description of a remote control method, device, electronic device, storage medium, and program product provided in this application, with reference to the accompanying drawings.

[0041] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0042] The terms "first" and "second," etc., used in the specification and drawings of this application are used to distinguish different objects or to distinguish different treatments of the same object, rather than to describe a specific order of objects.

[0043] Furthermore, the terms "comprising" and "having," and any variations thereof, used in the description of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0044] It should be noted that in the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0045] To facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish the same or similar items with essentially the same function and effect. Those skilled in the art can understand that the terms "first" and "second" are not intended to limit the quantity or execution order.

[0046] In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0047] The remote control method, device, electronic device, storage medium, and program products provided in this application can be applied to scenarios involving remote control of personal terminals. Specifically, they can be applied to scenarios where an intelligent agent built on an intranet platform controls a personal terminal.

[0048] With the rapid development of artificial intelligence technology, AI agents have been widely used in the field of remotely controlling computer terminals. Especially in scenarios where AI agents on intranet platforms control personal terminals, enterprises need to enable AI agents on the intranet platform to securely and in real-time control employees' personal computer terminals to perform various office automation tasks. Specifically, a Model Context Protocol Server (MCPServer) can be started on the terminal first, and then the AI ​​agent can be configured to connect to the MCP Server. Users can then issue control commands to the AI ​​agent, allowing the AI ​​agent to interact with the terminal multiple times based on these commands, invoking the corresponding local tools on the MCP Server to control the terminal.

[0049] However, the above solution has the following drawbacks: 1) Before calling the corresponding local tool on the MCP Server to control the terminal, the AI ​​agent needs to perform multiple signaling interactions with the terminal. Therefore, it may take a long time to wait before controlling the terminal, resulting in a long time consumption for controlling the terminal, thus affecting the efficiency of terminal control; 2) The existing solution of directly starting the MCP Server on the personal terminal has security risks. The personal terminal needs to expose an accessible MCP Server interface, which increases security and privacy risks.

[0050] To address the aforementioned technical problems, this application provides a remote control method, device, electronic device, storage medium, and program product. The approach involves constructing a three-layer architecture: an AI agent, a target server, and a target terminal. The target server is communicatively connected to the AI ​​agent and to a client running on the target terminal. The target server receives a task request message from the AI ​​agent, requesting the target terminal to perform a target operation using a target tool. This message indicates the tool identifier corresponding to the target tool and the operation parameters corresponding to the target operation. Based on the tool identifier and operation parameters, the target server generates a target task, which is a task to control the target terminal to perform the target operation using the target tool. The target task is then placed in a target task queue corresponding to the target terminal, and tasks from the target task queue are pushed to the client. When the target server receives a task request message from the AI ​​agent requesting the target terminal to use the target tool to perform a target operation, it can generate a target task to control the target terminal to perform the target operation based on the tool identifier and operation parameters in the task request message. In this way, the target server can put the target task into the target task queue corresponding to the target terminal, and directly push the task in the target task queue to the client running on the target terminal. That is, it can directly push the target task to the client without having to perform multiple signaling interactions with the target terminal. Therefore, it can reduce the waiting time before controlling the target terminal, thereby reducing the time spent controlling the target terminal and improving the efficiency of terminal control.

[0051] Furthermore, since the target terminal runs a client that is connected to the target server, the target terminal does not need to expose the MCP Server interface to the target server (and / or AI agent), thus protecting the security and privacy of the target terminal.

[0052] The embodiments provided in this application will now be described in detail with reference to the accompanying drawings.

[0053] The remote control method provided in this application can be applied to, for example... Figure 1 The application environment shown. For example... Figure 1 As shown, the application environment includes: AI agent 101, server 102, and terminal 103. AI agent 101 is interconnected with server 102, and a client runs on terminal 103, which is also interconnected with server 102.

[0054] In some embodiments, the AI ​​agent 101 can be deployed in an electronic device and generate instructions related to the information input by the user in the electronic device (e.g., voice input, text input, etc.).

[0055] In some embodiments, server 102 may be a server cluster consisting of multiple servers, a single server, a computer, or a processor or processing chip in a server or computer. This application does not limit the specific device form of server 102. Figure 1 The example shown is server 102 as a single server.

[0056] In some embodiments, terminal 103 can be a device with wireless transceiver capabilities, such as a mobile phone, tablet computer, wearable device, in-vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc. This application embodiment does not limit the specific device form of terminal 103. Figure 1 The example shown is a personal computer terminal, 103.

[0057] In some embodiments, the AI ​​agent 101 communicates with the server 102 using the MCP protocol. The server 102 is responsible for task distribution, tool management, and communication coordination, and exposes the MCP protocol interface for the AI ​​agent 101 to call. The terminal 103 receives tasks and performs computer operations, and the execution results are sent back to the server 102 via HTTP POST. This architecture design eliminates the need for the terminal 103 to run and expose an MCP server; it only needs to run a lightweight client (e.g., an Endpoint Agent) and interact with the AI ​​agent 101 through an intermediate layer (i.e., server 102), effectively protecting the security and privacy of the terminal 103.

[0058] It should be noted that the system architecture described in the embodiments of this application is for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and does not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0059] See Figure 2 This is a flowchart illustrating a remote control method provided in an embodiment of this application. The method is applied to a target server, which is communicatively connected to an AI agent and to a client running on a target terminal. Figure 2 As shown, the remote control method provided in this application specifically includes the following steps S201~S203.

[0060] S201, The target server receives a task request message from the AI ​​agent.

[0061] In this embodiment of the application, the task request message is used to request the target terminal to use the target tool to perform the target operation. The task request message indicates the tool identifier corresponding to the target tool and the operation parameters corresponding to the target operation.

[0062] In some embodiments of this application, the target server may include the following modules: A framework module: providing tool registration, management, and invocation interfaces. This module automatically registers all tool functions, providing standardized tool invocation interfaces for AI agents. A task distribution module: responsible for managing task queues and asynchronous task processing. This module maintains the task queue (endpoint_task_queues) for each terminal, using asyncio.Queue to implement asynchronous task management, supporting concurrent processing by clients on multiple terminals. It also maintains a result callback dictionary (_results) to store Future objects containing task execution results. A communication module: including a task push interface (endpoint_sse) to push tasks to terminals via connection; a result receiving interface (endpoint_result) to receive execution results returned by terminals; and a heartbeat maintenance mechanism to periodically send heartbeat packets to maintain the connection.

[0063] In some embodiments of this application, the target server may connect to the AI ​​agent based on at least one of the following protocols: MCP protocol, gRPC protocol, JSON-RPC protocol, etc.

[0064] In some instances, when the target server connects to the AI ​​agent based on the MCP protocol, the AI ​​agent communicates with the target server via an SSE connection and sends task requests using the tool interface defined by the MCP protocol. The target server processes the MCP protocol requests through the FastMCP framework, which configures the SSE path. The target server then mounts the FastMCP SSE application to the route.

[0065] In some instances, where the target server connects to the AI ​​agent based on the gRPC protocol, the target server can use the gRPC framework to define the service interface and message format for tool calls; the target server implements the gRPC server to provide the service interface for tool calls; the AI ​​agent implements the gRPC client to call the gRPC service interface to send tool call requests; gRPC uses Protocol Buffers for serialization and supports multiple programming languages.

[0066] As can be understood, in this example, gRPC is a high-performance RPC framework that supports streaming and bidirectional communication; it defines service interfaces through protobuf, implementing standardized tool call interfaces; gRPC supports multiple transport protocols (HTTP / 2, TCP, etc.), providing flexible communication methods.

[0067] In some instances, where the target server connects to the AI ​​agent based on the JSON-RPC protocol, the target server can use the JSON-RPC 2.0 protocol to define the request and response formats for tool calls; the target server implements a JSON-RPC server to handle tool call requests; the AI ​​agent sends a JSON-RPC request, including the method name (tool name) and parameters; and the target server executes the tool call.

[0068] As can be understood, in this example, JSON-RPC is a lightweight RPC protocol that is simple to implement and easy to understand; JSON-RPC supports batch requests, allowing multiple tools to be invoked at once; JSON-RPC is a standard protocol with implementation libraries in various programming languages.

[0069] In some embodiments of this application, the target server may connect to the client based on at least one of the following protocols: SSE communication protocol, WebSocket protocol, WebRTC protocol, etc.

[0070] In some instances, where the target server connects to the client based on the SSE communication protocol, the target server can establish an SSE connection with the client, enabling the client to connect to the target server's task push interface.

[0071] In some instances, when the target server connects to the client based on the WebRTC protocol, the target server establishes a WebRTC signaling server to handle the signaling exchange for connection establishment; the client on the target terminal establishes a point-to-point connection through WebRTC to achieve real-time bidirectional communication; the target server pushes tasks to the client on the terminal through the WebRTC data channel, and the client on the target terminal sends back the execution results through the same data channel; WebRTC supports audio and video transmission and can be expanded to support screen sharing and remote desktop functions in the future.

[0072] As can be understood, in this example, WebRTC is a peer-to-peer communication protocol, which can reduce server load; WebRTC supports low-latency communication, making it suitable for scenarios with high real-time requirements; and WebRTC supports NAT traversal, enabling connections to be established in complex network environments.

[0073] In some embodiments of this application, the target tool may include, but is not limited to, at least one of the following: mouse operation tool, keyboard operation tool, screen operation tool, etc., wherein the mouse operation tool may include, but is not limited to, at least one of the following: move_to, click, drag_to, scroll, etc., the keyboard operation tool may include, but is not limited to, at least one of the following: typewrite, hotkeys, etc., and the screen operation tool may include, but is not limited to, screenshot, etc.

[0074] In some embodiments of this application, the task request message may or may not carry a tool identifier and operation parameters.

[0075] In some embodiments of this application, the tool identifier may include, but is not limited to, at least one of the following: tool name, tool ID, etc.

[0076] In some embodiments of this application, the AI ​​agent described above can call predefined tool interfaces (and / or proxy functions) according to the user's natural language instructions to send task request messages to the target server.

[0077] S202. The target server generates the target task based on the tool identifier and operation parameters.

[0078] In this embodiment of the application, the aforementioned target task is the task of controlling the target terminal to perform target operations based on the target tool.

[0079] In some embodiments of this application, the target server may first generate a task identifier based on the tool identifier and operation parameters, and then generate a target task based on the task identifier, tool identifier, and operation parameters.

[0080] In some embodiments of this application, combined with Figure 2 ,like Figure 3 As shown, prior to S202 above, another remote control method provided in this application embodiment may also include S200 below, and S202 above can be specifically implemented by S2021 and S2022 below.

[0081] S200, The target server obtains the client identifier corresponding to the client.

[0082] In some instances, the aforementioned client identifiers may include, but are not limited to, Endpoint ID, client name, client ID, and client address. The Endpoint ID can be a persistent ID generated or read when the target terminal starts the client.

[0083] In some instances, the target server may pre-store the client identifier, allowing it to directly obtain the client identifier; alternatively, the target server may obtain the client identifier from the target terminal (or AI agent).

[0084] For example, the target server can extract the client identifier using the `push_and_wait` function, thereby obtaining the client identifier.

[0085] S2021. The target server generates a target task identifier based on the client identifier, tool identifier, and tool parameters corresponding to the client.

[0086] In this embodiment of the application, the target task identifier is used to indicate the target task.

[0087] In some instances, the target server can perform hash operations on the client identifier, tool identifier, and tool parameters separately, and then perform target operations on the calculated hash values ​​to obtain the final hash value. The target server can then use a preset correspondence to generate a string corresponding to the final hash value to obtain the target task identifier.

[0088] In other instances, the target server can input client identifiers, tool identifiers, and tool parameters into a pre-trained model, which then runs using these identifiers to output the target task identifier. This model can be trained using multiple sample client identifiers, multiple sample tool identifiers, and multiple sample tool parameters.

[0089] It should be noted that the training method for the preset model can be found in the specific descriptions in the relevant technologies, and will not be repeated here in the embodiments of this application.

[0090] S2022. The target server encapsulates the target task based on the client identifier, tool identifier, tool parameters, and target task identifier.

[0091] In some instances, the target server can encapsulate the client identifier, tool identifier, tool parameters, and target task identifier into a payload object to obtain the target task.

[0092] It should be noted that for the description of how the target server encapsulates the client identifier, tool identifier, tool parameters and target task identifier into a payload object, please refer to the specific description in the relevant technology, and the embodiments of this application will not be repeated here.

[0093] Thus, it can be seen that since the target server can obtain the client identifier corresponding to the client and generate a target task identifier based on the client identifier, tool identifier, and tool parameters, after encapsulation, the target task can be associated with the client identifier, target task identifier, tool identifier, and tool parameters. Therefore, on the one hand, the target server can accurately add the target task to the target task queue corresponding to the target terminal, so as to accurately push the target task to the target terminal. On the other hand, it can enable the target terminal to know the association between the target task and the target task identifier. In subsequent steps, the target terminal can associate the execution result with the target task identifier and report it to the target server, so that the target server can accurately know the task corresponding to the execution result, which is convenient for the target server to accurately know the execution result of the target task.

[0094] S203. The target server puts the target task into the target task queue corresponding to the target terminal and pushes the task in the target task queue to the client.

[0095] In some embodiments of this application, the target task queue may include, but is not limited to, at least one of the following: an asynchronous task queue based on asyncio.Queue, a task queue based on RabbitMQ, a task queue based on Redis, or a task queue based on Kafka.

[0096] In some embodiments of this application, when the target task queue includes an asynchronous task queue based on asyncio.Queue, the target server can use an asynchronous method to put the target task into the target task queue: `awaitendpoint_task_queues[endpoint_id].put(task_object)`.

[0097] Understandably, in this example, this application can use an in-memory asyncio.Queue, avoiding the overhead of message persistence and improving performance.

[0098] In this example, the target server can look up the corresponding task queue based on the client identifier: `queue = endpoint_task_queues[endpoint_id]`, and create it automatically if it does not exist.

[0099] In this example, the target server can push tasks in real time through an asynchronous generator (event_generator) connected via SSE, using the standard SSE format: `yield f"event: task\ndata: {json.dumps(task)}\n\n"`; the client on the target terminal receives the task events through the SSE connection, parses the SSE event format, and extracts the event_type and data fields. When the event_type is "task", it parses the JSON data in the data field to extract information from the target task.

[0100] In other embodiments of this application, when the target task queue includes at least one of a RabbitMQ-based task queue, a Redis-based task queue, and a Kafka-based task queue, the target server can send the target task to the target task queue, and the client of each terminal subscribes to the task queue corresponding to each terminal, so that the client of the target terminal can consume tasks from the target task queue.

[0101] Understandably, in this example, the target task queue supports distributed deployment, enabling task distribution across machines; the target task queue supports message persistence, ensuring tasks are not lost due to server restarts; and the target task queue supports a message acknowledgment mechanism, guaranteeing the reliability of task execution.

[0102] In some embodiments of this application, when the target server and the client of the target terminal are connected via the SSE communication protocol, the target server can push tasks to the client through the SSE connection, which is implemented using an asynchronous generator.

[0103] In this example, the client on the target terminal can connect to the SSE interface of the target server via an HTTP GET request: `GET / ai-test / computeruse / endpoint / sse?endpoint_id={endpoint_id}`; the target server returns a StreamingResponse, setting the media_type to "text / event-stream"; the SSE connection remains in a persistent state, enabling real-time push capabilities.

[0104] In this example, the task push uses the SSE standard format: `event: task\ndata: {JSON}\n\n`; the task data is in JSON format.

[0105] In some embodiments of this application, when the target server and the client of the target terminal are connected via the WebSocket protocol, the target server can establish a WebSocket server to listen for WebSocket connection requests, and the client of the target terminal can establish a WebSocket client to connect to the WebSocket server of the target server, so that the target server can push tasks (e.g., target tasks) to the client of the target terminal through the WebSocket connection.

[0106] It is understandable that WebSocket is a full-duplex communication protocol, which does not require the return of results via a separate HTTP POST like SSE; it can achieve real-time bidirectional communication, with task distribution and result return occurring within the same connection; the WebSocket connection persistence mechanism is similar to SSE, both of which can achieve stable long-term connections.

[0107] In some embodiments of this application, combined with Figure 2 ,like Figure 4 As shown, after S203 above, another remote control method provided in this application embodiment may also include S204 and S205 as described below.

[0108] S204. The target server receives the target task execution result from the client.

[0109] In this embodiment of the application, the execution result of the target task is the execution result of the target task.

[0110] In some instances, after executing the target task, the target terminal can send the result of the target task execution back to the target server via a separate HTTP POST request; the URL of the HTTP POST request is: `POST / ai-test / computeruse / endpoint / result`.

[0111] The request body is in JSON format and may contain at least one of the fields endpoint_id, task_id, and result. For example, it may contain endpoint_id, task_id, and result fields as: `{"endpoint_id": endpoint_id, "task_id": task_id, "result": result}`.

[0112] In some instances, the target server uses `asyncio.wait_for(future, timeout=10.0)` to asynchronously wait for the task execution result; when the target server's client sends back the result, it looks up the corresponding target Future object based on the task_id (i.e., the target task identifier): `future = _results.get(task_id)`; it sets the result of the target Future object: `future.set_result(result)`, triggering the coroutine waiting for the result to continue execution.

[0113] Understandably, this application uses Future objects to implement asynchronous waiting, avoiding the overhead of polling query results and improving efficiency.

[0114] S205. The target server sends the target task execution result to the AI ​​agent.

[0115] Thus, it can be seen that when the target server receives the execution result of the target task, it can send the execution result to the AI ​​agent so that the AI ​​agent can know the execution result. In this way, the AI ​​agent can provide timely feedback to the user based on the execution result, so that the user can perform the next operation. Therefore, the convenience of terminal operation can be improved.

[0116] In some instances, the execution result of the aforementioned target task includes a target task identifier, which is used to indicate the target task. Combined with... Figure 4 ,like Figure 5 As shown, prior to S205 above, another remote control method provided in this application embodiment may further include S301 and S302 as described below.

[0117] S301. The target server determines the target Future object associated with the target task identifier from at least one Future object based on the target task identifier.

[0118] S302. The target server stores the execution result of the target task in the target Future object.

[0119] Optionally, the target server can receive the result in the `endpoint_result` function, extract the `task_id` field (i.e., the target task identifier), use `asyncio.Lock` to ensure thread safety, and look up the corresponding Future object (i.e., the target Future object) from the result callback dictionary based on the `task_id`. Check if the target Future object exists and is not yet completed, and set the result of the target Future object. After setting the result of the target Future object, trigger the coroutine waiting for the result (`asyncio.wait_for(future, timeout=10.0)` in the `push_and_wait` function) to continue execution, thereby obtaining the target task execution result and sending the target task execution result.

[0120] Thus, it can be seen that since the target server can determine the target Future object associated with the target task identifier from at least one Future object based on the target task identifier, and store the target task execution result in the target Future object, this facilitates the target server to implement asynchronous task processing. Therefore, the target server can accurately obtain the target task execution result.

[0121] In some instances, combined Figure 5 ,like Figure 6 As shown, prior to S301 above, another remote control method provided in this application embodiment may also include the following S300.

[0122] S300: The target server creates a target Future object and establishes an association between the target Future object and the target task identifier.

[0123] Optionally, when the target task is enqueued (i.e., the target task queue), a target Future object is created: `future = syncio.get_event_loop().create_future()`; the target Future object is stored in the result callback dictionary: `_results[task_id] = future`, using asyncio.Lock to ensure thread safety; the key of the result callback dictionary is task_id (i.e., the target task identifier), and the value is the target Future object, realizing a precise association between the task and the result.

[0124] Thus, since the target server can create a target Future object and establish an association between the target Future object and the target task identifier, it can facilitate the target server to implement asynchronous task processing. Therefore, the target server can accurately obtain the execution result of the target task.

[0125] In some instances, combined Figure 5 ,like Figure 7 As shown, after S302 above, another remote control method provided in this application embodiment may also include S303 below.

[0126] S303, The target server deletes the target Future object from at least one Future object.

[0127] It is understandable that after the target task is completed, the target server can delete the target Future object: `_results.pop(task_id, None)`, thereby avoiding memory leaks and ensuring system stability.

[0128] Thus, it can be seen that since the target server can delete the target Future object after sending the target task execution result to the AI ​​agent, that is, after the target task is completed, the target server's storage space can be saved.

[0129] In some embodiments of this application, combined with Figure 2 ,like Figure 8 As shown, after S203 above, another remote control method provided in this application embodiment may also include S206 below.

[0130] S206. If the target server sends a heartbeat packet to the client when the target task queue is empty for a period of time greater than or equal to the time threshold.

[0131] In some instances, the target server can use `asyncio.wait_for(queue.get(), timeout=10)` to wait for tasks when the target task queue is empty; after the timeout, it sends a heartbeat to keep the connection active; and sends a heartbeat every 10 seconds to avoid connection timeout.

[0132] Therefore, since the target server can send a heartbeat packet to the client when the target task queue is empty for a period of time greater than or equal to a time threshold (i.e., when no task has been pushed to the client for a relatively long time), it can avoid the situation where the target server or the target terminal disconnects due to the long period of no task being pushed to the client. This avoids the need to re-establish the connection between the two before the target server pushes a task to the client again, thus improving the efficiency and reliability of task push.

[0133] This application provides a remote control method. A target server receives a task request message from an AI agent, requesting a target terminal to perform a target operation using a target tool. The task request message indicates the tool identifier corresponding to the target tool and the operation parameters corresponding to the target operation. Based on the tool identifier and operation parameters, a target task is generated, which is a task to control the target terminal to perform the target operation using the target tool. The target task is then placed in a target task queue corresponding to the target terminal, and the task in the target task queue is pushed to the client. Since the target server, upon receiving the task request message from the AI ​​agent requesting the target terminal to perform a target operation using a target tool, can generate a target task based on the tool identifier and operation parameters in the task request message, the target server can place the target task in the target task queue corresponding to the target terminal and directly push the task in the target task queue to the client running on the target terminal. This means the target task can be directly pushed to the client without multiple signaling interactions with the target terminal, thus reducing the waiting time before controlling the target terminal and improving the efficiency of terminal control.

[0134] Furthermore, since the target terminal runs a client that is connected to the target server, the target terminal does not need to expose the MCP Server interface to the target server (and / or AI agent), thus protecting the security and privacy of the target terminal.

[0135] Furthermore, this application can achieve standardized communication between AI agents and tools through the aforementioned protocol, improving the system's versatility and scalability; through a real-time communication mechanism, it realizes real-time task distribution and result feedback, reducing task execution latency and improving response speed; through an asynchronous task processing mechanism, it supports concurrent task processing by clients on multiple terminals, making full use of system resources and improving overall performance; by providing a complete system architecture and methodology, it enables AI agents to remotely control computer terminals through natural language commands, making it suitable for a variety of different application scenarios.

[0136] In some embodiments of this application, the task request message includes a request to call a target proxy function, which is a proxy function of the tool function of the target tool. (In conjunction with...) Figure 2 ,like Figure 9 As shown, prior to S202 above, another remote control method provided in this application embodiment may further include the following S401.

[0137] S401. The target server determines the tool identifier and operation parameters based on the call request and the target proxy function.

[0138] Thus, since the target server can determine the tool identifier and operation parameters based on the call request and the target proxy function, it is not necessary to deploy specific tool functions on the target server. Therefore, code redundancy can be reduced and storage space on the target server can be saved.

[0139] In some instances, combined Figure 9 ,like Figure 10 As shown, prior to S401 above, another remote control method provided in this application embodiment may further include S402 and S403 as described below.

[0140] S402, The target server receives the tool registry from the client.

[0141] In this embodiment of the application, the above-mentioned tool registry includes tool functions of at least one tool in the target terminal, and the at least one tool includes the target tool.

[0142] S403. The target server calls the make_proxy function for each tool's utility function to generate a proxy function for each tool's utility function, resulting in at least one proxy function.

[0143] In this embodiment of the application, the at least one proxy function mentioned above includes a target proxy function.

[0144] It is understandable that the utility functions of the target terminal are automatically registered in the tool registry (tool_registry) by being marked with the `@register_tool` decorator; the same set of tool code can be used on both the target server (through proxy functions) and the target terminal (directly called); the proxy function of the target server is generated by the `make_proxy` function, and the proxy function internally calls the `push_and_wait` function to distribute the task to the client of the target terminal; the client of the target terminal directly calls the utility functions in the tool registry to perform operations.

[0145] Thus, since the target server can receive the tool registry from the client and call the make_proxy function for the tool function of each tool in the tool registry to generate the proxy function of the tool function of each tool, it is not necessary to deploy the specific tool function of any tool in the target terminal on the target server. Therefore, code redundancy can be reduced and storage space on the target server can be saved.

[0146] It is understandable that the utility function code in this application only needs to be written once. Code reuse is achieved through the decorator mechanism, eliminating the need to write a separate set on the target server and the target terminal client, thereby avoiding code duplication and reducing development and maintenance costs.

[0147] See Figure 11 This is a flowchart illustrating another remote control method provided in this application embodiment, applied to a target terminal, which runs a client that communicates with a target server. Figure 11 As shown, the remote control method provided in this application specifically includes the following steps S501~S502.

[0148] S501, The target terminal receives the target task from the target server through the client.

[0149] In this embodiment of the application, the aforementioned target task is the task of controlling the target terminal to perform target operations based on the target tool.

[0150] S502. The target terminal performs target operations based on the target task and the target tool.

[0151] In some embodiments of this application, the target terminal can learn the tool identifier and operation parameters of the target tool according to the target task, and find the corresponding tool function (i.e. the tool function of the target tool) from the tool registry according to the tool identifier, and call the tool function of the target tool using the operation parameters to perform the target operation based on the target tool.

[0152] In some embodiments of this application, combined with Figure 11 ,like Figure 12 As shown, the above S502 can be specifically implemented through the following S5021~S5023.

[0153] S5021. The target terminal determines the tool identifier corresponding to the target tool and the operation parameters corresponding to the target operation based on the target task.

[0154] S5022. The target terminal retrieves the tool functions of the target tool from the tool registry based on the tool identifier.

[0155] In this embodiment of the application, the above-mentioned tool registry includes tool functions of at least one tool in the target terminal.

[0156] S5023. The target terminal uses the operation parameters to call the tool function of the target tool to perform the target operation using the target tool.

[0157] Thus, since the target terminal can accurately determine the tool function of the target tool from the tool registry based on the tool identifier, and accurately call the tool function of the target tool using the operation parameters, the target terminal can accurately execute the target operation using the target tool, thereby improving the accuracy of terminal operation.

[0158] In some instances, combined Figure 12 ,like Figure 13 As shown, prior to step 5021 above, another remote control method provided in this application embodiment may further include steps S601 to S602 as described below.

[0159] S601, The target terminal registers at least one tool into the tool registry.

[0160] In this embodiment of the application, the above-mentioned tool registry is used to create proxy functions for tool functions.

[0161] It should be noted that for instructions on registering at least one tool to the tool registry on the target terminal, please refer to the specific descriptions in related technologies, which will not be repeated here in the embodiments of this application.

[0162] S602. The target terminal uses the decorator to mark the tool functions in the tool registry and sends the marked tool registry to the target server.

[0163] Thus, since the target terminal can register at least one tool in the tool registry, mark the tool functions in the tool registry with decorators, and send the marked tool registry to the target server, the target server can create proxy functions for the tool functions of each tool. This eliminates the need to deploy the specific tool functions on the target server, thereby reducing code redundancy and saving storage space on the target server.

[0164] This application provides a remote control method in which a target terminal receives a target task from a target server through a client. This target task is a task to control the target terminal to perform a target operation based on a target tool. The target terminal then performs the target operation based on the target tool according to the target task. Since the target terminal can directly receive the target task from the target server through the client, without needing multiple signaling interactions with the target server, the target terminal can quickly perform the target operation based on the target tool according to the target task. Therefore, the waiting time before controlling the target terminal can be reduced, thereby reducing the time spent controlling the target terminal and improving the efficiency of terminal control.

[0165] Furthermore, since the target terminal runs a client that is connected to the target server, the target terminal does not need to expose the MCP Server interface to the target server (and / or AI agent), thus protecting the security and privacy of the target terminal.

[0166] Furthermore, this application can achieve standardized communication between AI agents and tools through the aforementioned protocol, improving the system's versatility and scalability; through a real-time communication mechanism, it realizes real-time task distribution and result feedback, reducing task execution latency and improving response speed; through an asynchronous task processing mechanism, it supports concurrent task processing by clients on multiple terminals, making full use of system resources and improving overall performance; by providing a complete system architecture and methodology, it enables AI agents to remotely control computer terminals through natural language commands, making it suitable for a variety of different application scenarios.

[0167] In some embodiments of this application, combined with Figure 11 ,like Figure 14 As shown, after S502 above, another remote control method provided in this application embodiment may also include S503 as described below.

[0168] S503: The target terminal sends the target task execution result to the target server through the client.

[0169] In this embodiment of the application, the execution result of the target task is the execution result of the target task.

[0170] Thus, since the target terminal can send the target task execution result to the target server through the client, the target server can know the target task execution result and promptly feed it back to the AI ​​agent. Therefore, the AI ​​agent can promptly provide feedback to the user based on the target task execution result, so that the user can perform the next operation. This improves the convenience of terminal operation.

[0171] In some embodiments of this application, combined with Figure 14 ,like Figure 15 As shown, the target task execution result includes a target task identifier, which is used to indicate the target task. Prior to S503, another remote control method provided in this application embodiment may further include the following S504.

[0172] S504. The target terminal determines the target task identifier based on the target task.

[0173] In some instances, the target terminal can parse the target task to obtain the target task identifier from the target task.

[0174] Thus, since the target terminal can determine the target task identifier based on the target task, in subsequent steps, the target terminal can associate the execution result with the target task identifier and report it to the target server, so that the target server can accurately know the task corresponding to the execution result. Therefore, it is convenient for the target server to accurately know the execution result of the target task.

[0175] The remote control method of this application embodiment is described below with reference to a specific example. The specific implementation process of this method is as follows: Figure 16 As shown.

[0176] Step 601: The target server receives a task request message from the AI ​​agent.

[0177] Step 602: The target server generates the target task based on the tool identifier and operation parameters.

[0178] Step 6031: The target server puts the target task into the target task queue corresponding to the target terminal.

[0179] Step 6032: The target server pushes the tasks in the target task queue to the client.

[0180] Step 604: The target terminal receives the target task from the target server through the client.

[0181] Step 605: The target terminal performs the target operation based on the target tool according to the target task.

[0182] Step 606: The target terminal sends the target task execution result to the target server through the client.

[0183] Step 607: The target server receives the target task execution result from the client.

[0184] Step 608: The target server sends the target task execution result to the AI ​​agent.

[0185] This application provides a remote control method. When the target server receives a task request message from an AI agent requesting the target terminal to use a target tool to perform a target operation, it can generate a target task to control the target terminal to perform the target operation based on the tool identifier and operation parameters in the task request message. In this way, the target server can put the target task into the target task queue corresponding to the target terminal, so as to directly push the task in the target task queue to the client running on the target terminal. That is, the target task can be directly pushed to the client without multiple signaling interactions with the target terminal. Therefore, the waiting time before controlling the target terminal can be reduced, thereby reducing the time spent controlling the target terminal and improving the efficiency of terminal control.

[0186] The following will illustrate another remote control method provided in the embodiments of this application with specific examples.

[0187] Assuming the target server and AI agent are connected via the SSE communication protocol, and the target server and the client of the target terminal are connected via the SSE communication protocol, another remote control method provided in this application includes the following steps: Step 1: System initialization.

[0188] 1.1. Target Server Initialization: Create a FastMCP instance, configure the SSE path and message path. Scan the tool directory and automatically register all tool functions to the MCP framework. Initialize the task queue dictionary (endpoint_task_queues) and result callback dictionary (_results). Create the target task queue, start the SSE server, and listen for task push requests.

[0189] 1.2. Target Terminal Client Initialization: When the target terminal's client program starts, it generates or reads a persistent Endpoint ID (e.g., the client identifier in the above embodiment). It establishes an SSE connection to the target server's task push interface. It registers the computer operating tools to the local tool registry. It starts a background SSE listening thread, waiting for tasks to be received.

[0190] Specifically, this application achieves real-time communication with SSE based on the MCP protocol through the following specific technical methods: (1) Establishment of MCP protocol and SSE connection: The AI ​​agent communicates with the target server through the SSE connection and sends task request messages using the tool interface defined by the MCP protocol; the target server processes the task request messages through the FastMCP framework, and the FastMCP framework configures the SSE path: `mcp = FastMCP(name="RedisStreamMCP",sse_path=" / ai-test / computeruse / sse",message_path=" / ai-test / computeruse / messages / ")`; the target server mounts the FastMCP SSE application to the route: `Mount(" / ", app=mcp.sse_app())`.

[0191] (2) Specific method for real-time task push via SSE: After the target server receives the MCP protocol tool call request from the AI ​​agent, it extracts the endpoint_id parameter through the `push_and_wait` function; it finds the corresponding target task queue based on the endpoint_id: `queue = endpoint_task_queues[endpoint_id]` (automatically created if it does not exist); it encapsulates the task into a payload object (i.e., the target task), containing endpoint_id, task_id, tool, and args, and puts it into the queue: `await endpoint_task_queues[endpoint_id].put(payload)`; it pushes the task in real time through the asynchronous generator (event_generator) connected via SSE, using the SSE standard format: `yield f"event: task\ndata:` {json.dumps(task)}\n\n"`;The target terminal's client receives task events via an SSE connection, parses the SSE event format, and extracts the event_type and data fields; when event_type is "task", it parses the JSON data in the data field and extracts task_id, tool, and args.

[0192] Step 2: Tool registration.

[0193] All utility functions are automatically registered via a decorator mechanism: A utility function is defined using the `@register_tool` decorator. Each utility function includes a detailed function signature and docstring. When the target server starts, it automatically scans the utility registry and creates a proxy function for each tool. These proxy functions are exposed to the AI ​​agent via the MCP protocol, standardizing the tool's calling interface.

[0194] Specifically, based on the MCP protocol, this application achieves automatic tool registration and proxy function generation through the following specific technical methods: (1) Implementation of the decorator mechanism: Define the `register_tool` decorator function, which takes the utility function as a parameter; the decorator function registers the utility function in the tool registry (tool_registry): `tool_registry[func.__name__] = func`; the utility function is marked with the `@register_tool` decorator and the registration is completed automatically.

[0195] (2) Specific method for generating proxy functions: When the target server starts, it iterates through all the tool functions in the tool registry: `for name in computer_tools.tool_registry:`; for each tool function, it calls the `make_proxy` function to create a proxy function; the proxy function keeps the signature of the original function through `inspect.signature` and keeps the documentation of the original function through `__doc__`; the proxy function is exposed to the AI ​​agent through the MCP protocol: `mcp.tool()(make_proxy(name))`, realizing the standardization of the tool call interface.

[0196] (3) Specific implementation of the push_and_wait function (a key link connecting MCP protocol calls and asynchronous task processing): - The `push_and_wait` function receives the tool name (tool_name) and parameters (args), extracts the endpoint_id from args: `endpoint_id = args.get("endpoint_id") or os.getenv("endpoint_id")`; generates a unique task ID: `task_id = str(uuid.uuid4())`; creates a target Future object: `future = asyncio.get_event_loop().create_future()`; uses asyncio.Lock to ensure thread safety, stores the target Future object in the result callback dictionary: `async with _results_lock: _results[task_id]= future`; encapsulates the task into a payload object, containing endpoint_id, task_id, tool, and args; uses an asynchronous method to put the task into a queue: `await` `endpoint_task_queues[endpoint_id].put(payload)`; use `asyncio.wait_for(future, timeout=10.0)` to asynchronously wait for the task execution result; after the task execution is completed, remove the target Future object from the result callback dictionary: `async with _results_lock:_results.pop(task_id, None)`.

[0197] (4) Implementation method of tool code reuse: The tool function is automatically registered in the tool registry (tool_registry) by marking it with the `@register_tool` decorator; the same set of tool code can be used on the target server (through proxy function) or on the client of the target terminal (direct call); the proxy function of the target server is generated by the `make_proxy` function, and the proxy function calls the `push_and_wait` function to distribute the task to the client of the target terminal; the client of the target terminal directly calls the tool function in the tool registry to perform the operation.

[0198] It is understandable that the tool code in this application only needs to be written once. Code reuse is achieved through the decorator mechanism, eliminating the need to write a separate set on the target server and the target terminal, thereby avoiding code duplication and reducing development and maintenance costs.

[0199] As can be seen from the above, this application, through a three-layer distributed architecture, allows the target terminal to run a client Endpoint Agent (not an MCP Server), eliminating the need to expose the MCP Server interface and protecting the security and privacy of the personal terminal. This application implements a complete technical solution for automatic tool registration and proxy function generation through a decorator mechanism and the `make_proxy` function. This application connects MCP protocol tool calls with asynchronous task processing through the `push_and_wait` function, converting MCP protocol calls into asynchronous task distribution. This application standardizes tool calls by maintaining the original function signatures and documentation, enabling AI agents to call different tools through a unified interface. This application avoids the problem of writing separate sets of tool code for the target server and the target terminal, as is present in existing technologies, through a tool code reuse mechanism.

[0200] Step 3: Task distribution process.

[0201] 3.1. AI Agent Initiates Task: Based on the user's natural language instructions, the AI ​​agent invokes the tool interface defined by the MCP protocol to send a task request (e.g., the task request message in the above embodiment) to the target server. The task request includes the tool name "tool" (e.g., the tool identifier in the above embodiment) and the tool parameters "args" (e.g., the operation parameters in the above embodiment).

[0202] 3.2. Target server receives task: The target server receives the task request from the AI ​​agent, extracts the endpoint_id, tool and args parameters, and generates a unique task ID (task_id), which can be the target task identifier in the above embodiment.

[0203] 3.3. Task Enqueueing: The target server encapsulates the task into a task object (e.g., the target task in the above embodiment), containing task_id, tool, args, and endpoint_id, and places it into the target task queue (endpoint_task_queues[endpoint_id]) of the client of the corresponding target terminal. At the same time, a target Future object is created and stored in the result callback dictionary (_results[task_id]) for subsequent result callbacks.

[0204] 3.4. SSE Push Task: The target server pushes the task to the corresponding client terminal via an SSE connection. The client terminal's SSE listening thread receives the task event, parses the task object, and extracts task_id, tool, and args.

[0205] Step 4: Task execution.

[0206] 4.1. Tool Invocation: The client on the target terminal searches for the corresponding tool function in the tool registry based on the received tool name, and invokes the tool function using the args parameter.

[0207] 4.2. Computer Operation Execution: Utility functions execute corresponding computer operations, such as mouse movement, keyboard input, and screen capture. After the operation is completed, the execution result is returned.

[0208] 4.3. Result Encapsulation: The client of the target terminal encapsulates the execution result into a result object (such as the target task execution result in the above embodiment), which includes the task_id and result fields.

[0209] Step 5: Return the results.

[0210] 5.1. HTTP POST Return: The client on the target terminal sends the result object to the result receiving interface (endpoint_result) of the target server via an HTTP POST request.

[0211] 5.2. Result Callback: After receiving the result, the target server looks up the corresponding target Future object from the result callback dictionary based on the task_id, sets the result into the target Future object, and triggers the coroutine waiting for the result to continue execution.

[0212] Specifically, after the client on the target terminal executes the target task, it sends the result back to the target server via a separate HTTP POST request. The URL of the HTTP POST request is: `POST / ai-test / computeruse / endpoint / result`. The request body is in JSON format and includes endpoint_id, task_id, and result fields: `{"endpoint_id": endpoint_id, "task_id": task_id, "result": result}`. The target server receives the result in the `endpoint_result` function, extracts the task_id field, uses asyncio.Lock to ensure thread safety, and looks up the corresponding target Future object from the result callback dictionary based on task_id: `async with _results_lock: future = _results.get(task_id)`. It then checks if the target Future object exists and is not yet completed, and sets the result of the target Future object: `if future and not future.done():`. `future.set_result(result)`: After setting the result of the target Future object, the coroutine waiting for the result (`asyncio.wait_for(future, timeout=10.0)` in the `push_and_wait` function) is triggered to continue execution.

[0213] It is understood that this application combines the MCP protocol with the SSE connection through the SSE path configuration of the FastMCP framework; this application achieves real-time bidirectional communication by pushing tasks through SSE and returning results via HTTP POST, thus overcoming the limitation of unidirectional communication through SSE.

[0214] 5.3. Return to AI Agent: The target server returns the results to the AI ​​agent, completing a full task execution process.

[0215] Furthermore, when the task queue is empty, use `asyncio.wait_for(queue.get(), timeout=10)` to wait for tasks; after the timeout, send a heartbeat packet: `yield "event: heartbeat\ndata: ping\n\n"`; send a heartbeat packet every 10 seconds to avoid connection timeout.

[0216] Furthermore, in the above steps, this application uses Python's asyncio.Queue to implement asynchronous task queue management, and the specific technical methods include: (1) Creation and management of task queues: Use Python's `collections.defaultdict` and `asyncio.Queue` to create a task queue dictionary: `endpoint_task_queues = defaultdict(asyncio.Queue)`; maintain an independent task queue for each terminal client, with the key being endpoint_id and the value being an asyncio.Queue object; when the terminal client connects for the first time, the corresponding task queue is automatically created: `queue = endpoint_task_queues[endpoint_id]`.

[0217] (2) Asynchronous queuing method for tasks: When the AI ​​agent calls the tool, the target server receives the task request, extracts the endpoint_id, tool and args parameters; generates a unique task ID: `task_id = str(uuid.uuid4())`; encapsulates the task into a task object (i.e. the target task): `{"task_id": task_id, "tool": tool, "args": args, "endpoint_id": endpoint_id}`; and puts the task into the queue using an asynchronous method: `awaitendpoint_task_queues[endpoint_id].put(task_object)`.

[0218] It is understandable that this application uses an in-memory asyncio.Queue to avoid the overhead of message persistence and improve performance.

[0219] (3) Asynchronous dequeueing and push method of tasks: The target server pushes tasks to the client of the target terminal through the SSE connection, which is implemented by an asynchronous generator; when the task queue is empty, a heartbeat packet is sent to maintain the connection: `yield "event:heartbeat\ndata: ping\n\n"`.

[0220] It is understandable that this application uses SSE to push tasks in real time, eliminating the need for client-side polling on the terminal, thus achieving real-time task distribution.

[0221] Asynchronous result callback mechanism based on Future objects: This application uses Python's Future objects to implement asynchronous result callbacks. Specific technical methods include: (1) Creation and storage of Future objects: When a task is enqueued, a Future object is created: `future = asyncio.get_event_loop().create_future()`; the Future object is stored in the result callback dictionary: `_results[task_id] = future`, and asyncio.Lock is used to ensure thread safety; the key of the result callback dictionary is task_id, and the value is a Future object, realizing the precise association between task and result.

[0222] (2) Implementation of asynchronous waiting for results: Use `asyncio.wait_for(future, timeout=10.0)` to asynchronously wait for the task execution result; when the client of the terminal sends back the result, look up the corresponding Future object according to the task_id: `future = _results.get(task_id)`; set the result of the Future object: `future.set_result(result)`, triggering the coroutine waiting for the result to continue execution.

[0223] Understandably, this application uses Future objects to implement asynchronous waiting, avoiding the overhead of polling query results and improving efficiency.

[0224] (3) Cleanup mechanism for result callbacks: After the task is completed, the Future object is removed from the result callback dictionary: `_results.pop(task_id, None)`; to avoid memory leaks and ensure system stability.

[0225] SSE-based real-time task push mechanism: This application uses SSE to implement real-time task push, and the specific technical methods include: (1) Establishment of SSE connection: The client of the target terminal connects to the SSE interface of the target server through an HTTP GET request: `GET / ai-test / computeruse / endpoint / sse?endpoint_id={endpoint_id}`; The target server returns a StreamingResponse and sets the media_type to "text / event-stream"; The SSE connection maintains a long connection state to realize real-time push capability.

[0226] (2) Real-time push format of tasks: Task push adopts SSE standard format: `event: task\ndata:{JSON}\n\n`; Task data adopts JSON format, including task_id, tool, args and endpoint_id; Heartbeat packet format: `event: heartbeat\ndata: ping\n\n`.

[0227] (3) Heartbeat maintenance mechanism: When the task queue is empty, use `asyncio.wait_for(queue.get(),timeout=10)` to wait for the task; after the timeout, send a heartbeat packet to keep the connection active; send a heartbeat packet every 10 seconds to avoid connection timeout.

[0228] As can be seen from the above, the main differences between the middleware implementation method used in this application and the prior art include: 1. Existing message queue methods require the terminal client to poll and query tasks. This application uses SSE to push tasks in real time, avoiding polling overhead and improving real-time performance.

[0229] 2. Existing RPC methods use a synchronous request-response model. This application uses asyncio.Queue and Future objects to implement asynchronous processing, which supports high concurrency and improves system performance.

[0230] 3. Existing message queue methods require message persistence. This application uses in-memory asyncio.Queue, avoiding I / O overhead and improving performance.

[0231] 4. Existing middleware methods rely on message order to ensure the association between tasks and results. This application achieves precise association through task_id and Future objects, and supports out-of-order result return.

[0232] Furthermore, this application adopts a distributed architecture, deploying the AI ​​agent, target server, and target terminal client separately. The target server, acting as an intermediate layer, is responsible for task distribution and result aggregation, supporting simultaneous connections from multiple AI agents and multiple terminal clients. Each terminal client maintains an independent task queue, ensuring no interference between them and achieving system scalability.

[0233] The implementation process of this application is illustrated below through a specific embodiment.

[0234] Example: An AI agent remotely controls a target terminal to open Notepad and input text.

[0235] Scenario description: The user sends the command "Open Notepad, type 'Hello World' and save" to the AI ​​agent.

[0236] Implementation steps: Step 10: AI Agent Understands Instructions: After receiving the user's natural language instructions, the AI ​​agent understands the intent of the instructions and plans the execution steps: 1. Open the Notepad application; 2. Type the text "Hello World" into Notepad; 3. Save the file.

[0237] Step 11: Calling the tool interfaces: The AI ​​agent calls the tool interfaces defined by the MCP protocol in sequence according to the plan: (1) To open the hotkeys tool, press the Win key to open the search box; (2) To open the typewrite tool, enter "notepad"; (3) To open the hotkeys tool, press the Enter key; (4) To open the screenshot tool, take a screenshot to confirm that Notepad is open; (5) To open the typewrite tool, enter "Hello World"; (6) To open the hotkeys tool, press Ctrl+S to save.

[0238] Step 12, Target Server Processes Tasks: After receiving each tool invocation request, the target server: 1. Extract endpoint_id, tool, and args parameters; 2. Generate a unique task_id; 3. Add the task to the task queue of the client on the corresponding terminal; 4. Create a Future object and wait for the result.

[0239] Step 13, Target terminal client executes task: After the SSE listening thread of the target terminal client receives the task: 1. Parse the task object and extract tool and args; 2. Locate the corresponding utility function in the utility registry; 3. Call utility functions to perform computer operations; 4. Return the execution result.

[0240] Step 14, Result Feedback: The client on the target terminal sends the execution result back to the target server via HTTP POST; after receiving the result, the target server sets the target Future object and returns the result to the AI ​​agent.

[0241] Step 15, Task Completion: After receiving the execution results from all tools, the AI ​​agent confirms the task is completed and provides feedback to the user.

[0242] In some embodiments of this application, the steps in the above embodiments can also be replaced by the following scheme, as follows: Alternative Solution 1: Use a polling mechanism to replace SSE for task reception.

[0243] In the mechanism for receiving tasks on the target terminal's client, an HTTP polling mechanism can be used to replace the SSE long connection to achieve task reception.

[0244] Specifically, the client of the target terminal periodically (e.g., every 1 second) sends an HTTP GET request to the target server to query whether there are any tasks to be executed; after receiving the request, the target server retrieves the task from the task queue of the corresponding terminal's client and returns it to the terminal's client; if there are no tasks to be executed, the target server returns an empty response, and the client of the target terminal continues to poll; after receiving the task, the client of the target terminal executes it, and after execution, sends the result back via HTTP POST.

[0245] It is understandable that the above polling mechanism is simple to implement and does not require maintaining a long connection; the polling interval can be adjusted according to actual needs to balance real-time performance and server load; the polling mechanism is suitable for scenarios with unstable network environments, and can automatically reconnect after the connection is lost. This alternative solution uses a polling mechanism to replace the SSE long connection, but it also achieves the functions of task distribution and result feedback. Although the real-time performance is slightly lower than SSE, it can still achieve the purpose of remotely controlling the computer terminal by the AI ​​agent, and is a technically equivalent alternative solution of this application.

[0246] Alternative Solution 2: Use the publish-subscribe pattern to replace the task queue for task distribution.

[0247] In the task distribution mechanism, the publish-subscribe (Pub-Sub) pattern can be used to replace the task queue to achieve task distribution.

[0248] Specifically, the target server publishes the task to a message topic, and each terminal's client subscribes to the corresponding topic; after receiving the task, the terminal's client executes it and publishes the result to the result topic; the target server subscribes to the result topic, receives the execution result, and matches the task request according to the task_id; the publish-subscribe pattern supports multiple subscribers and can achieve task broadcasting and load balancing.

[0249] It is understandable that the publish-subscribe pattern supports decoupling, meaning task publishers and subscribers do not need to communicate directly; it supports message filtering, allowing subscription to messages of interest based on conditions; and it supports expansion, making it easy to add new subscribers. This alternative uses the publish-subscribe pattern to replace the task queue, but still achieves task distribution and result feedback, supports concurrent processing by clients on multiple terminals, and is a technically equivalent alternative to this application.

[0250] Alternative Solution 3: Directly call the proxy function of the alternative tool via HTTP.

[0251] In the tool invocation mechanism, direct HTTP calls can be used to replace tool proxy functions to implement tool invocation.

[0252] Specifically, the target server provides an independent HTTP interface for each tool, such as ` / api / tools / {tool_name}`; the AI ​​agent directly calls the HTTP interface, sends a POST request containing the tool parameters; after receiving the HTTP request, the target server directly calls the corresponding tool function and returns the execution result; the tool function returns the execution result through the HTTP response, without needing to use a Future object callback.

[0253] It is understandable that HTTP calls are simple to implement, easy to understand and debug; HTTP calls support RESTful style and conform to Web API design specifications; HTTP calls support multiple clients and can be easily integrated into different systems. This alternative uses direct HTTP calls to replace tool proxy functions, but still achieves the function of tool calls. Although it lacks the standardization of the MCP protocol, it can still achieve the purpose of remotely controlling computer terminals by AI agents, and is a technically equivalent alternative to this application.

[0254] The technical solution presented in this application has good feasibility for implementation. Specific details are as follows: I. Production environment deployment of the technical solution.

[0255] The technical solution presented in this application can be deployed in a production environment, and the system operates stably with complete functionality. Production environment deployment includes: 1. MCP Server Layer: The target server is deployed in the cloud to provide stable service support. The server adopts a distributed architecture, supports concurrent client connections from multiple terminals, and has high availability and scalability.

[0256] 2. Terminal Client: A Windows version of the terminal client (EndpointAgent.exe) has been developed and released, which users can download and use directly. The program has functions such as automatic Endpoint ID generation, SSE connection management, and task execution. It supports system tray residency and is easy to use.

[0257] 3. Communication protocol support: The system fully supports the MCP protocol and SSE real-time communication, realizing standardized real-time communication between AI agents and terminal agents.

[0258] II. Integration with AI native application development platforms.

[0259] The technical solution presented in this application, when used in conjunction with the intelligent agent built on the AI ​​native application development platform, enables a complete application scenario.

[0260] Application method: 1. The user runs the EndpointAgent.exe program on the target terminal. After the program starts, it automatically generates and displays a unique Endpoint ID.

[0261] 2. Users configure the target server connection via SSE URL in the agent configuration of the AI ​​native application development platform.

[0262] The URL format can be: `http: / / server address / ai-test / computeruse / sse?endpoint_id=YOUR_ENDPOINT_ID`.

[0263] The YOUR_ENDPOINT_ID mentioned above is the Endpoint ID of the user's PC.

[0264] 3. After configuration, users can remotely control their personal PC using natural language commands through the platform's intelligent agent's chat interface, enabling computer operation functions such as mouse operation, keyboard input, and screen capture.

[0265] Application effects: Users do not need to understand the underlying technical details and can use it with simple configuration; complex computer operations can be completed through natural language commands, improving the user experience; it supports multi-terminal agent concurrency and can manage multiple terminal PCs at the same time; the real-time communication mechanism ensures the real-time performance and response speed of the operation.

[0266] III. Maturity and reliability of the technical solution.

[0267] 1. Technology Maturity: The technical solution in this application has been verified in a production environment, the system architecture is stable, and the technology implementation is mature. The MCP protocol, SSE real-time communication, and asyncio asynchronous processing technologies used are all mature and stable technical standards.

[0268] 2. System Reliability: The system possesses robust error handling mechanisms, heartbeat maintenance mechanisms, and task queue management mechanisms, ensuring system stability and reliability. The production environment operates stably without any major failures.

[0269] 3. Scalability: The system adopts a distributed architecture design, supports concurrent processing by multiple terminal agents, and can be horizontally expanded according to actual needs, thus possessing good scalability.

[0270] IV. Application Scenarios and Promotion Value.

[0271] The technical solution of this invention is applicable to the following application scenarios: 1. Remote control of personal PC terminal: Users can remotely control their personal PC terminal through AI intelligent agents to achieve automated operation and improve work efficiency.

[0272] 2. Enterprise Office Automation: Enterprise employees can remotely control office computers through AI intelligent agents to automate the execution of office tasks.

[0273] 3. IT Operations Automation: IT operations personnel can remotely control server terminals through AI agents to automate operations tasks.

[0274] 4. Education and Training: Education and training institutions can remotely control teaching terminals through AI intelligent agents to automate the execution of teaching tasks.

[0275] In some embodiments, this application also provides a remote control device. The remote control device may include one or more functional modules for implementing the remote control method of the above method embodiments.

[0276] For example, Figure 17 This is a schematic diagram illustrating the composition of a remote control device provided in an embodiment of this application. The device is applied to a target server, which is communicatively connected to an AI agent and to a client running on a target terminal. Figure 17 As shown, the remote control device 700 includes: a receiving module 701, a generating module 702, a joining module 703, and a sending module 704.

[0277] The system includes the following modules: a receiving module 701, which receives a task request message from the AI ​​agent, requesting the target terminal to perform a target operation using a target tool; the task request message indicates the tool identifier corresponding to the target tool and the operation parameters corresponding to the target operation; a generating module 702, which generates a target task based on the tool identifier and operation parameters, which controls the target terminal to perform the target operation using the target tool; an adding module 703, which adds the target task generated by the generating module 702 to the target task queue corresponding to the target terminal; and a sending module 704, which pushes the task object from the target task queue added by the adding module 703 to the client.

[0278] This application provides a remote control device. When the remote control device receives a task request message from an AI agent requesting a target terminal to perform a target operation using a target tool, it can generate a target task to control the target terminal to perform the target operation based on the tool identifier and operation parameters in the task request message. In this way, the remote control device can put the target task into the target task queue corresponding to the target terminal, so as to directly push the task in the target task queue to the client running on the target terminal. That is, it can directly push the target task to the client without having to perform multiple signaling interactions with the target terminal. Therefore, it can reduce the waiting time before controlling the target terminal, thereby reducing the time spent controlling the target terminal and improving the efficiency of terminal control.

[0279] In some embodiments, another remote control device 700 provided in this application may further include: an acquisition module, configured to acquire a client identifier corresponding to the client before the generation module 702 generates a target task based on a tool identifier and tool parameters. Specifically, the generation module 702 is configured to generate a target task identifier based on the client identifier, tool identifier, and tool parameters corresponding to the client, the target task identifier being used to indicate the target task; and to encapsulate the target task based on the client identifier, tool identifier, tool parameters, and target task identifier.

[0280] Thus, it can be seen that since the remote control device can obtain the client identifier corresponding to the client and generate a target task identifier based on the client identifier, tool identifier, and tool parameters, after encapsulation, the target task can be associated with the client identifier, target task identifier, tool identifier, and tool parameters. Therefore, on the one hand, the remote control device can accurately add the target task to the target task queue corresponding to the target terminal, so as to accurately push the target task to the target terminal. On the other hand, it can enable the target terminal to know the association between the target task and the target task identifier. In subsequent steps, the target terminal can report the execution result and the target task identifier to the remote control device, so that the remote control device can accurately know the task corresponding to the execution result, which is convenient for the remote control device to accurately know the execution result of the target task.

[0281] In other embodiments, the task request message includes a call request for a target proxy function, which is a proxy function of a tool function of the target tool. Another remote control device 700 provided in this application embodiment may further include: a determining module, configured to determine the tool identifier and operation parameters based on the call request and the target proxy function before the generation module 702 generates the target task based on the tool identifier and tool parameters.

[0282] Thus, since the remote control device can determine the tool identifier and operation parameters based on the call request and the target proxy function, it is not necessary to deploy specific tool functions in the remote control device. Therefore, code redundancy can be reduced and storage space of the remote control device can be saved.

[0283] In some other embodiments, the receiving module 701 is further configured to receive a tool registry from a client before the determining module determines the tool identifier and operation parameters based on the call request and the target proxy function. The tool registry includes tool functions of at least one tool in the target terminal, and the at least one tool includes the target tool. The generating module 702 is further configured to call the `make_proxy` function on the tool function of each tool to generate a proxy function for each tool, resulting in at least one proxy function that includes the target proxy function.

[0284] Thus, since the remote control device can receive the tool registry from the client and call the make_proxy function for the tool function of each tool in the tool registry to generate the proxy function of the tool function of each tool, it is not necessary to deploy the specific tool function of any tool in the target terminal in the remote control device. Therefore, code redundancy can be reduced and storage space of the remote control device can be saved.

[0285] In some other embodiments, the sending module 704 is further configured to send a heartbeat packet to the client after pushing the task object in the target task queue to the client, provided that the time during which the target task queue is empty is greater than or equal to a time threshold.

[0286] Thus, since the remote control device can send a heartbeat packet to the client when the target task queue is empty for a period of time greater than or equal to a time threshold (i.e., when no task has been pushed to the client for a relatively long time), it can avoid the situation where the connection between the remote control device and the target terminal is disconnected due to the long period of no task being pushed to the client. This avoids the need to re-establish the connection between the two before the remote control device pushes the task to the client again, thereby improving the efficiency and reliability of task pushing.

[0287] In some other embodiments, the receiving module 701 is further configured to receive the execution result of the target task from the client after the sending module 704 pushes the task object in the target task queue to the client. The execution result of the target task is the execution result of the target task. The sending module 704 is further configured to send the execution result of the target task to the AI ​​agent.

[0288] Thus, it can be seen that when the remote control device receives the execution result of the target task, it can send the execution result to the AI ​​agent so that the AI ​​agent can know the execution result. In this way, the AI ​​agent can provide timely feedback to the user based on the execution result, so that the user can take the next step of operation. Therefore, the convenience of terminal operation can be improved.

[0289] In some embodiments, the target task execution result includes a target task identifier, which is used to indicate the target task. Another remote control device 700 provided in this application embodiment may further include: a determining module and a storage module; a sending module, used to determine a target Future object associated with the target task identifier from at least one Future object based on the target task identifier before the sending module 704 sends the target task execution result to the AI ​​agent; and a storage module, used to store the target task execution result in the target Future object determined by the determining module.

[0290] Thus, it can be seen that since the remote control device can determine the target Future object associated with the target task identifier from at least one Future object based on the target task identifier, and store the target task execution result in the target Future object, this facilitates the remote control device to implement asynchronous task processing. Therefore, the remote control device can accurately obtain the target task execution result.

[0291] In some other embodiments, the remote control device 700 provided in this application may further include: a creation module and an establishment module. The creation module is used to create a target Future object before the determining module determines a target Future object associated with the target task identifier from at least one Future object based on the target task identifier. The establishment module is used to establish the association between the target Future object created by the creation module and the target task identifier.

[0292] Thus, since the remote control device can create a target Future object and establish an association between the target Future object and the target task identifier, it can facilitate the remote control device to perform asynchronous task processing. Therefore, the remote control device can accurately obtain the target task execution result.

[0293] In some other embodiments, the remote control device 700 provided in this application may further include: a deletion module, used to delete the target Future object from at least one Future object after the storage module stores the target task execution result in the target Future object.

[0294] Thus, it can be seen that since the remote control device can delete the target Future object after sending the target task execution result to the AI ​​agent, that is, after the target task is completed, the storage space of the remote control device can be saved.

[0295] For example, Figure 18 This is a schematic diagram illustrating another remote control device provided in an embodiment of this application, applied to a target terminal. The target terminal runs a client that communicates with a target server. Figure 18 As shown, the remote control device 800 includes a receiving module 801 and an execution module 802.

[0296] The receiving module 801 is used to receive a target task from the target server through a client. This target task is a task to control the target terminal to perform a target operation based on the target tool. The execution module 802 is used to execute the target operation based on the target tool according to the target task received by the receiving module 801.

[0297] This application provides a remote control device. Since the remote control device can directly receive the target terminal from the target server through the client to perform the target operation based on the target tool, without having to perform multiple signaling interactions with the target server, the remote control device can quickly perform the target operation based on the target tool according to the target task. Therefore, the waiting time before controlling the target terminal can be reduced, thereby reducing the time spent controlling the target terminal and improving the efficiency of terminal control.

[0298] In some embodiments, the execution module 802 is specifically configured to determine, based on the target task, the tool identifier corresponding to the target tool and the operation parameters corresponding to the target operation; and based on the tool identifier, determine the tool function of the target tool from the tool registry, which includes the tool function of at least one tool in the target terminal; and call the tool function of the target tool using the operation parameters to perform the target operation using the target tool.

[0299] Thus, since the remote control device can accurately determine the tool function of the target tool from the tool registry based on the tool identifier, and accurately call the tool function of the target tool using the operation parameters, the target terminal can accurately perform the target operation using the target tool, thereby improving the accuracy of terminal control.

[0300] In other embodiments, the execution module 802 is further configured to register at least one tool in a tool registry before determining the tool identifier corresponding to the target tool and the operation parameters corresponding to the target operation based on the target task, and to mark the tool functions in the tool registry using decorators. Another remote control device provided in this application embodiment may further include: a sending module, configured to send the tool registry marked by the execution module 802 to a target server; wherein the tool registry is used to create proxy functions for tool functions.

[0301] Thus, it can be seen that since the remote control device can register at least one tool to the tool registry, mark the tool functions in the tool registry with decorators, and send the marked tool registry to the target server, the target server can create proxy functions for the tool functions of each tool. This eliminates the need to deploy the specific tool functions on the target server, thereby reducing code redundancy and saving storage space on the target server.

[0302] In some other embodiments, the remote control device provided in this application may further include: a sending module, used to send the target task execution result to the target server through a client after the execution module 802 performs the target operation based on the target tool according to the target task, the target task execution result being the execution result of the target task.

[0303] Thus, since the remote control device can send the execution result of the target task to the target server through the client, the target server can know the execution result of the target task and promptly feed back the execution result to the AI ​​agent. Therefore, the AI ​​agent can promptly provide feedback to the user based on the execution result of the target task, so that the user can perform the next step of operation. Therefore, the convenience of terminal operation can be improved.

[0304] In some other embodiments, the target task execution result includes a target task identifier, which is used to indicate the target task. The execution module 802 is further configured to determine the target task identifier based on the target task before the sending module sends the target task execution result to the target server via the client.

[0305] Thus, since the remote control device can determine the target task identifier based on the target task, in subsequent steps, the remote control device can associate the execution result with the target task identifier and report it to the target server, so that the target server can accurately know the task corresponding to the execution result. Therefore, it is convenient for the target server to accurately know the execution result of the target task.

[0306] In the case where the functions of the integrated modules described above are implemented in hardware, this application provides a schematic diagram of the composition of an electronic device as described in the above embodiments. For example... Figure 19 As shown, the electronic device 900 includes: a processor 902, a communication interface 903, and a bus 904. Optionally, the electronic device 900 may also include a memory 901.

[0307] Processor 902 may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 902 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field-programmable gate array, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 902 may also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0308] The communication interface 903 is used to connect to other devices via a communication network. This communication network can be Ethernet, wireless access network, wireless local area network (WLAN), etc.

[0309] The memory 901 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0310] In one possible implementation, the memory 901 can exist independently of the processor 902. The memory 901 can be connected to the processor 902 via a bus 904 and is used to store instructions or program code. When the processor 902 calls and executes the instructions or program code stored in the memory 901, it can implement the remote control method provided in the embodiments of this application.

[0311] In another possible implementation, the memory 901 can also be integrated with the processor 902.

[0312] The 904 bus can be an Extended Industry Standard Architecture (EISA) bus, etc. The 904 bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 9 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0313] Through the above description of the implementation methods, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the service calling device can be divided into different functional modules to complete all or part of the functions described above.

[0314] This application also provides a computer-readable storage medium. All or part of the processes in the above method embodiments can be executed by computer instructions instructing related hardware. The program can be stored in the aforementioned computer-readable storage medium, and when executed, it can include the processes of the above method embodiments. The computer-readable storage medium can be any of the foregoing embodiments or memory. The aforementioned computer-readable storage medium can also be an external storage device of the aforementioned service invocation device, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the aforementioned service invocation device. Further, the aforementioned computer-readable storage medium can include both internal storage units of the aforementioned service invocation device and external storage devices. The aforementioned computer-readable storage medium is used to store the aforementioned computer program and other programs and data required by the aforementioned service invocation device. The aforementioned computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0315] This application also provides a computer program product, which includes a computer program that, when run on a computer, causes the computer to perform any of the remote control methods provided in the above embodiments.

[0316] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method of teleoperation, the method comprising: The application is applied to a target server, the target server is in communication connection with an artificial intelligence (AI) agent, the target server is in communication connection with a client running on a target terminal, and the remote control method comprises the following steps: receiving a task request message from the AI agent, the task request message being used to request the target terminal to execute a target operation by using a target tool, the task request message indicating a tool identifier corresponding to the target tool and an operation parameter corresponding to the target operation; generating a target task according to the tool identifier and the operation parameter, the target task being a task of controlling the target terminal to execute the target operation based on the target tool; putting the target task into a target task queue corresponding to the target terminal, and pushing a task in the target task queue to the client.

2. The teleoperation method of claim 1, wherein, Before the step of generating the target task according to the tool identifier and the tool parameter, the method further comprises the following steps: obtaining a client identifier corresponding to the client; the step of generating the target task according to the tool identifier and the tool parameter comprises the following steps: generating a target task identifier according to the client identifier corresponding to the client, the tool identifier and the tool parameter, the target task identifier being used to indicate the target task; encapsulating the client identifier, the tool identifier, the tool parameter and the target task identifier to obtain the target task.

3. The tele-manipulation method of claim 1, wherein, The task request message comprises a call request of a target proxy function, and the target proxy function is a proxy function of a tool function of the target tool; Before the step of generating the target task according to the tool identifier and the tool parameter, the method further comprises the following steps: determining the tool identifier and the operation parameter according to the call request and the target proxy function.

4. The tele-manipulation method of claim 3, wherein, Before the step of determining the tool identifier and the operation parameter according to the call request and the target proxy function, the method further comprises the following steps: receiving a tool registry from the client, the tool registry comprising tool functions of at least one tool in the target terminal, and the at least one tool comprising the target tool; calling a make_proxy function for the tool function of each tool to generate a proxy function of the tool function of each tool, and obtaining at least one proxy function, wherein the at least one proxy function comprises the target proxy function.

5. The tele-manipulation method of claim 1, wherein, After the step of pushing the task object in the target task queue to the client, the method further comprises the following steps: when a time when the target task queue is empty is greater than or equal to a time threshold, sending a heartbeat packet to the client.

6. The tele-manipulation method of claim 1, wherein, After the step of pushing the task object in the target task queue to the client, the method further comprises the following steps: receiving a target task execution result from the client, the target task execution result being an execution result of the target task; sending the target task execution result to the AI agent.

7. The tele-manipulation method of claim 6, wherein, The target task execution result comprises a target task identifier, and the target task identifier is used to indicate the target task; Before the step of sending the target task execution result to the AI agent, the method further comprises the following steps: determining, according to the target task identifier, a target Future object associated with the target task identifier from at least one Future object; storing the target task execution result in the target Future object.

8. The tele-manipulation method of claim 7, wherein, Before the determining, according to the target task identifier, a target Future object associated with the target task identifier from at least one Future object, the method further comprises: creating the target Future object and establishing an association between the target Future object and the target task identifier.

9. The tele-manipulation method of claim 7, wherein, After the storing, the target task execution result in the target Future object, the method further comprises: deleting the target Future object from the at least one Future object.

10. A remote control method characterized by comprising: applied to a target terminal, the target terminal running a client, the client being in communication connection with a target server, the remote control method comprising: receiving, by the client, a target task from the target server, the target task being a task of controlling the target terminal to perform a target operation based on a target tool; performing the target operation based on the target tool according to the target task.

11. The tele-manipulation method of claim 10, wherein, The performing the target operation based on the target tool according to the target task comprises: determining a tool identifier corresponding to the target tool and an operation parameter corresponding to the target operation according to the target task; determining a tool function of the target tool from a tool registry according to the tool identifier, the tool registry including tool functions of at least one tool in the target terminal; invoking the tool function of the target tool using the operation parameter to perform the target operation using the target tool.

12. The tele-manipulation method of claim 11, wherein, Before the determining a tool identifier corresponding to the target tool and an operation parameter corresponding to the target operation according to the target task, the method further comprises: registering the at least one tool in the tool registry; marking tool functions in the tool registry using a decorator and sending the marked tool registry to the target server; wherein the tool registry is used to create a proxy function of a tool function.

13. The tele-manipulation method of claim 10, wherein, After the performing the target operation based on the target tool according to the target task, the method further comprises: sending, by the client, a target task execution result to the target server, the target task execution result being an execution result of the target task.

14. The tele-manipulation method of claim 13, wherein, The target task execution result includes a target task identifier, the target task identifier being used to indicate the target task; Before the sending, by the client, a target task execution result to the target server, the method further comprises: determining the target task identifier according to the target task.

15. A tele-manipulation device, characterized by applied to a target server, the target server being in communication connection with an AI intelligent agent, the target server being in communication connection with a client running on a target terminal, the remote control device comprising: The receiving module is configured to receive a task request message from the AI agent, the task request message being used to request the target terminal to perform a target operation using a target tool, the task request message indicating a tool identifier corresponding to the target tool and an operation parameter corresponding to the target operation; The generating module is configured to generate a target task according to the tool identifier and the operation parameter, the target task being a task of controlling the target terminal to perform the target operation based on the target tool; The joining module is configured to put the target task generated by the generating module into a target task queue corresponding to the target terminal; The sending module is configured to push, to the client, a task object in the target task queue after the target task is added by the joining module.

16. A tele-manipulation device, characterized by The remote control device is applied to a target terminal, and a client is run on the target terminal, the client being in communication connection with a target server. The receiving module is configured to receive, by the client, a target task from the target server, the target task being a task of controlling the target terminal to perform a target operation based on a target tool; The executing module is configured to perform the target operation based on the target tool according to the target task received by the receiving module.

17. An electronic device, comprising: The computer readable storage medium comprises computer execution instructions, when the computer execution instructions are run on a computer, causing the computer to execute the remote control method in any one of claims 1 to 9, or execute the remote control method in any one of claims 10 to 14.

18. A computer-readable storage medium, characterized in that, The computer program product comprises a computer program, when the computer program is run on an electronic device, causing the electronic device to execute the remote control method in any one of claims 1 to 9, or execute the remote control method in any one of claims 10 to 14.

19. A computer program product, characterised in that, ​

Images