Method for collaboration of agentic managers

Abstract

A first proxy manager and a second proxy manager collaborate to respond to user requests. The first proxy manager receives the user request on the originating device and processes it using a first artificial intelligence (AI) model to generate a sequence of sub-requests. The first proxy manager then sends one or more sub-requests to a second proxy manager on the target device. The second proxy manager processes one or more sub-requests using a second AI model on the target device and sends the application output from the target device to the first proxy manager. Based on the application output, the first proxy manager generates a response to the user request on the originating device.

Description

[Technical Field]

[0001] Embodiments of the present invention relate to an agent system that supports artificial intelligence (AI) agents on edge devices. [Previous Technology]

[0002] Agent-based artificial intelligence (AI) systems are autonomous and goal-oriented, possessing the ability to make decisions based on predetermined goals and learned experiences. AI agents can utilize various AI models to communicate with humans and complete tasks. By leveraging diverse AI models, agent-based AI systems can perceive their environment, make informed decisions, interact naturally with humans, and autonomously perform complex tasks without step-by-step human input. Agent-based AI systems are capable of operating effectively in various fields and applications.

[0003] The AI ​​models used in agent-based AI systems may include machine learning models, deep learning models, natural language processing models, etc. Many of these models require significant memory space and computing resources. Typically, large language models (LLMs) may be stored in the cloud and made accessible to users remotely via a network. Cloud-based agent-based AI systems introduce latency, thus affecting real-time responsiveness, especially in time-sensitive or interactive tasks. Furthermore, using cloud-based systems may raise privacy and data security issues due to the transmission and remote processing of sensitive user data. However, edge devices are limited by memory size and computing resources. Therefore, providing agent-based AI systems on edge devices is a challenge. [Summary of the Invention]

[0004] According to one embodiment, a method for a collaborative proxy manager is provided. A first proxy manager receives a user request on an originating device and processes the user request using a first AI model to generate a sequence of sub-requests. The first proxy manager sends one or more sub-requests to a second proxy manager on a target device. The second proxy manager processes the one or more sub-requests using a second AI model on the target device and sends application output from the target device to the first proxy manager. Based on the application output, the first proxy manager generates a response to the user request on the originating device.

[0005] In one embodiment, each sub-request is accompanied by a session identifier that identifies the user session initiated by the user request. In one embodiment, the user request includes identification of a target device. In one embodiment, the originating device detects user activity of the target device in the vicinity of the originating device. In another embodiment, the target device receives a notification sent from the originating device and extracts the one or more sub-requests from the notification. In one embodiment, the originating device sends multiple sub-requests to multiple proxy managers on multiple target devices for processing. In one embodiment, the originating device uses a first AI model to determine whether each sub-request is executed by the originating device or the target device, and determines the execution order of the sub-requests based on dependencies in the sub-requests.

[0006] In another embodiment, a method is provided for runtime updates of application information by an agent manager on an edge device. The agent manager requests application information when the application runs. The application tracks information change times, indicating the time when the application received the latest update to the application information. The agent manager compares or obtains a comparison between the information change times and a database update time. The database update time indicates the latest time the application information was updated in a database on the edge device. The agent manager retrieves the application information from the application and the database based on which of the information change times and the database update time is more recent. The agent manager invokes the application to generate output. The application information includes one or more of the following: functions, application interfaces (APIs), and internal application data.

[0007] In one embodiment, the application maintains information change time and database update time, and compares the information change time with the database update time. In another embodiment, the proxy manager maintains the database update time and compares the database update time with the information change time obtained from the application. In one embodiment, the proxy manager requests application information when the application becomes the foreground application. In one embodiment, the proxy manager requests application information when the application starts running.

[0008] In another embodiment, the agent manager performs an on-the-fly summarization method on an edge device. The agent manager initiates a prompting session in response to a user request, wherein the prompting session includes multi-turn prompt-response exchanges between the agent manager and an inference artificial intelligence (AI) model on the edge device. The agent manager sends a prompt to the summarization AI model on the edge device to summarize the prompt, and receives a summary of the prompt from the summarization AI model. The agent manager sends the summary to the inference AI model to generate an action plan, and invokes an application based on the action plan to generate a response to the user request.

[0009] In one embodiment, the agent manager continues the prompting session by replacing each subsequent prompt in the prompting session with a corresponding summary summarizing the subsequent prompts. In one embodiment, the agent manager replaces a portion of the prompts in the prompting session with a corresponding summary. In one embodiment, the agent manager restarts a new prompting session by using a summary of the new prompt and past prompts accumulated in the prompting session as the initial prompt for a given AI model. In one embodiment, the agent manager uses the AI ​​model to generate a prompting session summary summarizing all prompts in the prompting session and stores the prompt summary in a database on an edge device. In one embodiment, the agent manager uses the prompting session summary to remember user preferences. In one embodiment, the agent manager uses the prompting session summary to remember past actions. In one embodiment, the inference AI model generates multiple action plans to trigger multiple applications or services on multiple devices.

[0010] Other aspects and features will become apparent to those skilled in the art upon review of the following description of specific embodiments and the accompanying drawings.

Implementation Method

[0012] Numerous specific details are set forth in the following description. However, it should be understood that embodiments of the invention can be carried out without these specific details. In other instances, well-known circuits, structures, and techniques have not been shown in detail so as not to obscure the understanding of this description. However, those skilled in the art will understand that the invention can be carried out without these specific details. Those skilled in the art will be able to implement appropriate functions without excessive experimentation based on the included description. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0013] In the following description, the term "agent manager" refers to a software application that can make autonomous decisions based on available and inferred information to drive other applications ("applications") or services. The term "agent application" (hereinafter referred to as "application") refers to a software application that can be instructed and / or orchestrated by the agent manager and take actions to provide services accessible to users, other applications, software, and / or systems. Although the term "application" or "application application" is used throughout the disclosure, the methods and systems described herein are not limited to applications on a device. In some embodiments, the methods and systems described herein are applicable to network services provided by cloud service providers, services on a device (e.g., system services, embedded services), etc. The term "cloud" refers to a remote system consisting of server computers, storage, and software that provides services to edge devices via a network (e.g., the Internet). The term "edge device" (hereinafter referred to as "device") refers to a device located at the boundary between a local network and a wide area network (e.g., the Internet) and providing access to the wide area network. Non-limiting examples of edge devices include smartphones, wearable devices, laptops, personal computers, Internet of Things (IoT) devices, navigation devices, infotainment devices, robotic devices, smart appliances, smart lights / switches, etc. The term "AI model" (hereinafter referred to as "model") used herein includes, but is not limited to: machine learning models, deep learning models, custom learning models, natural language processing models, large language models (LLMs), multimodal models, neural networks and their variants, etc. The terms "cloud AI model" or "cloud model" refer to AI models in the cloud, and "edge AI model" or "edge model" refers to AI models installed on edge devices. The term "edge node" (hereinafter referred to as "node") used herein includes virtual machines (VMs) and physical devices such as edge devices. A system may include multiple nodes, which may be VMs, edge devices, or a combination of both.

[0014] In one embodiment, the proxy framework (“framework”) may be deployed on a distributed proxy system comprising multiple nodes. The components of the framework (“framework components”) may be distributed across multiple nodes. One of the framework components is a proxy manager on one of the nodes to coordinate the operations of the other framework components.

[0015] In one embodiment, a distributed agent system may include multiple interactive peripherals in one or more network-connected devices. In one embodiment, different framework components may run on different VMs in one or more devices. In one embodiment, a distributed agent system may include multiple agent managers sharing the same model service and / or the same database service. In one embodiment, these multiple agent managers may be multiple agent manager instances instantiated from the same agent manager definition. In one embodiment, an agent manager may operate (i.e., invoke, start, etc.) applications and / or services located on multiple VMs and / or devices. In another embodiment, an agent manager may invoke services in the cloud and applications and / or services located on multiple VMs and / or devices.

[0016] The agent manager works with applications in an "agent-like" manner because they can make autonomous decisions to achieve a given goal, such as a goal given by a user, another application, or another device. Autonomous decisions can be based on learned data, metadata, pre-configured data, combinations of these data, etc. In one embodiment, the agent manager uses an AI model to perform AI operations. In one embodiment, one or more applications may also use an AI model to perform AI operations.

[0017] Taking smart home as an example. A distributed agent system can support multiple interactive peripherals, such as smart microphones and speakers, televisions, wearable devices, smartphones, cameras, IoT devices, etc., from which a user can issue requests. In embodiments where the distributed agent system is distributed across multiple devices, the agent manager and system functions can run on a router, the model service and database service can run on a home server, and applications can run on smart TVs or IoT devices to control door locks, thermostats, lights, etc. In one embodiment, the distributed agent system can support multiple agent managers running on multiple devices, for example, one agent manager running on a smart TV, another agent manager running on a tablet, and another agent manager running on a refrigerator. In one embodiment, multiple agent managers can share the same model service and / or database service. The model service and / or database service can run on the same device or VM, and at least one or more agent managers can run on another device or VM. In a smart home, the agent manager on a smartphone can operate applications on other devices; non-limiting examples include smart TVs, tablets, IoT devices, etc.

[0018] As another example, a distributed agent system can support multiple interactive peripherals in a vehicle, such as multiple displays. For example, there might be one display for the driver, another for the front passenger, and two displays for the rear passengers. In one embodiment, the smart cockpit device in the vehicle can run an agent manager and applications on one VM, and a model service and a database service on another VM. In one embodiment, the agent manager on a smartphone can operate applications on the smart cockpit device. Alternatively, the agent manager on the smart cockpit device can operate applications on a smartphone. In one embodiment, multiple agent managers can run on multiple devices and VMs in the vehicle.

[0019] The proxy framework described herein is deployed on a proxy system. Therefore, in the following description, the terms "proxy framework" and "proxy system" are used interchangeably. A distributed proxy system is a proxy system that includes multiple nodes. Unless otherwise stated, the methods described below can be applied to distributed proxy systems deployed on multiple edge nodes and proxy systems deployed on devices.

[0020] Figure 1 is a block diagram showing a proxy framework 105 (“Framework 105”) according to one embodiment. Non-limiting examples of framework components in Framework 105 include a proxy manager 180, a proxy application (“Application 150”), a model service 160, a database service 170, and one or more interactive peripherals 190. In one embodiment, Framework 105 may include multiple proxy managers 180, multiple applications 150, and / or multiple interactive peripherals 190. Proxy manager 180 is a proxy management application that operates to manage application 150 and interact with model service 160, database service 170, interactive peripherals 190, and users. Proxy manager 180 may also access applications and / or AI models in the cloud.

[0021] Model service 160 manages AI models in framework 105. These AI models are installed on one or more devices and are therefore referred to as edge models 164. Agent manager 180 and some applications 150 can access edge models 164. Edge models 164 may include base models, low-rank adaptation (LoRa) models, ControlNet models, and other additional models. Each model is described by corresponding model metadata, which can be stored in database 173 and / or retrieval augmented generation (RAG) database 172 for fast searching. Database 173 can be searched by keywords or other means. RAG database 172 is also referred to as vector embedding database or embedding database. Vector embedding (also called "embedding") is a numerical representation of the semantics of stored information. Embedding database can efficiently and accurately search for semantically similar information. Embedding is usually a high-dimensional vector, but not limited to it, encoding the semantic context and relationships of information.

[0022] Database service 170 manages database 173 and RAG database 172. In one embodiment, model metadata may be stored in RAG database 172 for vector embedding search (also known as "similarity search") and similarity ranking. Similarity ranking refers to ranking search results based on similarity according to search criteria, such as searching for a target model that meets the requirements of application 150. Model service 160 can automatically set the target model of application 150 according to the model requirements indicated in the application metadata. Application metadata describes the characteristics of application 150 and the requirements of the model used by application 150. Application metadata may be converted into vector embeddings by database service 170 and stored in RAG database 172. In one embodiment, application metadata describes the action requests that a given application can accept. Application metadata may further specify the specific model used by a given application, or specify the requirements of the model used by a given application. In some embodiments, application metadata may also describe one or more rules or prompts used by agent manager 180 to invoke a given application.

[0023] In one embodiment, the agent manager 180 includes an action engine 181, a prompting engine 182, and a context engine 184, all of which are coordinated by a logic core 185. The agent manager 180 interacts with the application 150, the model service 160, and the database service 170. The agent manager 180 also interacts with one or more users through one or more interactive peripherals 190. The agent manager 180 can access the edge model 164. In some embodiments, the agent manager 180 can also access system functions 110, which provide built-in system functions of the device in which the agent manager 180 resides. Non-limiting built-in system functions include services such as time, location, device manufacturer information, device ID information (e.g., phone number), device settings (e.g., font size), and device control functions (e.g., airplane mode). System functions 110 differ from the application 150, which are built-in system functions, while the application 150 is an independently developed capability.

[0024] In one embodiment, each interactive peripheral 190 is an I / O peripheral device capable of interacting with a user and / or the environment. The interactive peripheral 190 provides various forms of I / O for the user to interact with framework components. The operation of the interactive peripheral 190 may be managed by an I / O manager 194. For example, the interactive peripheral 190 may receive user input via touch, voice, text, and / or similar means. Non-limiting examples of the interactive peripheral 190 may include cameras, sensors, displays, speakers, microphones, IoT devices, robots, etc. The interactive peripheral 190 also generates output to the user and / or other I / O devices. In some embodiments, the interactive peripheral 190 may include an IoT device with a service agent (e.g., software, firmware, and / or hardware components) installed, wherein the service agent may be controlled by an agent manager 180. The interactive peripheral 190 may support one or more of the following: a graphical user interface (GUI) 191, a voice user interface (VUI) 192, a sensing interface 193, and / or other I / O interfaces. The GUI 191 may provide graphical icons or links on the display for the user to select and generate graphical output for the user to view. The VUI 192 may provide speech-to-text functionality (e.g., Automatic Speech Recognition (ASR)) and text-to-speech (TTS) functionality to convert the user's voice input into text and text output into speech. The sensing interface 193 may include touch sensors to sense the user's touch, a camera to detect the user's gestures, etc.

[0025] Framework 105 provides an agent experience to edge device users in a user-intuitive way. Agent manager 180 may utilize one or more edge models 164 for natural language processing, speech recognition, and speech generation. In one embodiment, agent manager 180 may be invoked by a user-triggered phrase, such as "hi there".

[0026] In one embodiment, the framework 105 is deployed on multiple nodes, including devices, virtual machines (VMs) on one or more devices, or a combination of both. The devices in the framework 105 are connected via a network.

[0027] Figure 2 is a block diagram illustrating the interaction between framework components according to one embodiment. When a user request is received, the interaction peripheral 190 forwards the user request to the agent manager 180. The user request may specify a task. The agent manager 180 (more specifically, the context engine 184) sends a context request to the database service 170 to obtain contextual information about the user request, such as the identity of one or more applications for the requested service. The database service 170 performs a similarity search in the RAG database 172 based on the similarity between stored application metadata and phrases in the user request. In one embodiment, the contextual information generated from the similarity search includes local information and / or user preference information, which may be used by the agent manager 180 to prompt one of the edge models 164, referred to as the target model. The contextual information can improve the quality and accuracy of the response generated by the target model, thereby enhancing the user experience. In one embodiment, the contextual information may identify one or more applications 150 as target applications to provide the service requested by the user.

[0028] Upon receiving context information, the agent manager 180 (more specifically, the prompting engine 182) sends a prompt to the agent target model, wherein the prompt contains context information. For example, the prompt may include a request for an action plan. The target model generates a response containing the action plan, indicating that the agent manager 180 may send an action request to the target application. The agent manager 180 (more specifically, the action engine 181) sends the action request to the target application. The target application executes the action and returns the action result to the agent manager 180. In one embodiment, the target application may use the database service 170 and / or the model service 160 to respond to the action request. In some cases, the agent manager 180 may send additional action requests to one or more applications based on the action plan. The agent manager 180 may send the action result to the user via the interactive peripheral 190 to obtain further input or confirmation from the user. When the user requests the completion of a specified task, the agent manager 180 sends an output to the user indicating the completion of the task. In one embodiment, the operation of the agent manager 180 is coordinated by the logic core 185.

[0029] In one embodiment, communication between the agent manager 180 and the application 150 is bidirectional. The agent manager 180 requests the target application to perform an action, and the target application sends the action result to the agent manager 180. For example, the action could be ordering a hamburger, and the action result could be a list of hamburgers provided by a food ordering application. This list can be provided to the agent manager 180 as the action result, and the agent manager 180 can query one or more AI models, online resources, and / or databases on the device to supplement the list with relevant information (e.g., nutrition and / or price), and then generate output for the user. In some cases, the action result returned by the target application to the agent manager 180 may be an indication of "success" or "failure" of the food order. When executing an action request, the target application can use one or more AI models to generate the action result. In some cases, the target application can generate output without using an AI model.

[0030] As previously mentioned, framework components can be distributed across multiple devices and / or multiple virtual machines (VMs). Before describing the management of framework components in a distributed framework, it is helpful to first explain the terms "user request" and "user session." A user request is a request sent by a user to the agent manager 180 to perform a task. A user session is a session that begins when a user sends a request to the agent manager 180 to perform a task and ends when the task is completed. As a non-limiting example, upon receiving a user request, the agent manager 180 may act to process the user request, access models, access databases, request the target application to perform actions, and output a response to the user. A user session may include one or more iterations of interaction between the user and the agent manager 180. For example, the agent manager 180 may ask the user for clarification, and the user may provide feedback to the agent manager 180.

[0031] Referring to Figure 1, in one embodiment, a plurality of interactive peripherals 190 may reside on a plurality of corresponding devices. I / O managers 194 and I / O interfaces (e.g., GUI 191, VUI 192, sensor interface 193, etc.) supported by the interactive peripherals 190 may run on one or more VMs on the same device. In one embodiment, an agent manager 180 may process user requests received by the plurality of interactive peripherals 190 in one of the following operating modes: a first mode is sequential session execution, and a second mode is concurrent session execution. In the sequential session execution mode, the agent manager 180 processes one user session at a time, regardless of whether the session request comes from the same or different interactive peripherals 190. Incoming session requests are stored in a first-in-first-out (FIFO) queue. The agent manager 180 initiates processing of the next session request only after the currently active session has completed. In the concurrent session execution mode, the agent manager 180 allows multiple user sessions to be processed simultaneously. Although user sessions can run in parallel, the agent manager 180 maintains a FIFO queue for action requests for the same application 150. This ensures that such action requests are executed sequentially in the order they are received, thereby maintaining logical consistency and integrity in interactions with each application 150.

[0032] Referring to Figure 1, in one embodiment, a plurality of applications 150 may reside on a plurality of nodes. Applications 150 on the same device may run on one or more VMs. In one embodiment, agent manager 180 may operate applications 150 on a local device (i.e., the same device as agent manager 180) and applications 150 on a remote device (i.e., a different device than agent manager 180). Agent manager 180 and applications 150 may run on the same VM or different VMs. Agent manager 180 and applications 150 may run on the same operating system (OS) or different OSs.

[0033] As a non-limiting example, in a smart home environment, an agent manager on a smartphone can control applications hosted on external devices (e.g., smart TVs, tablets, and / or IoT devices). As another non-limiting example, in an automotive environment, an agent manager on a smartphone can operate applications in a smart cockpit system, and vice versa.

[0034] Framework 105 supports the following coordination operations to enable agent manager 180 to operate applications hosted on remote nodes in a distributed system. The term "remote node" in a distributed system refers to an edge node (e.g., a device or virtual machine) that is different from the node where agent manager 180 resides. In one embodiment, database service 170 maintains application metadata and node identifiers (e.g., device ID or network address). The node identifier identifies the node where the application authorized for invocation by agent manager 180 resides. If a node identifier is specified by user request, agent manager 180 restricts its application discovery to the specified node. If no node is specified, agent manager 180 may prompt the user for such information. Alternatively, agent manager 180 may continue searching for applicable application 150 across all known nodes. If multiple candidate applications are found, agent manager 180 selects one of the application 150 based on an internal selection strategy, user-defined preferences, or by requesting user confirmation.

[0035] Before initiating a remote application search, the agent manager 180 verifies the operational availability of the target node. If the user-specified node is currently unavailable, the agent manager 180 provides a response to the user indicating the unavailability of that node. If the user does not specify a node, only currently available nodes are included in the search scope.

[0036] When an action request is sent to a target application hosted on a remote node, the agent manager 180 transmits the action request along with the metadata of the remote node to the local agent. The local agent then forwards the action request to the remote agent located on the remote node, and the remote agent then routes the action request to the target application.

[0037] Figure 3 illustrates a peer-to-peer communication configuration of a distributed agent system 300 according to one embodiment. In this embodiment, components of framework 105 (Figure 1) reside on multiple nodes (i.e., multiple devices and / or virtual machines). In Figure 3, each block with rounded corners represents a node. The framework components include one or more interactive peripherals 190, applications 150, agent managers 180, model services 160, and database services 170. These framework components communicate with each other through their respective available data communication channels. The selection and configuration of these channels depend on the relative positions of the interactive framework components and the execution environment.

[0038] In one embodiment, when two framework components are deployed on two different devices, communication between the two framework components can be established through available wired or wireless data channels, including but not limited to TCP / IP over Ethernet, Wi-Fi, Bluetooth, or similar protocols. In one embodiment, when two framework components are deployed on two different virtual machines on the same device, cross-virtual machine data channels can be used to facilitate communication between virtual machines; for example, by configuring the virtual machines to use the same network, such as TCP / IP over a bridged network, Network Address Translation (NAT), or equivalent mechanisms.

[0039] In one embodiment, a proxy (shown as "P" in the circle in Figure 3) is provided in each node where one or more framework components reside. The proxy operates to pass inter-component communication data to each other. The proxy abstracts and encapsulates the details of the underlying communication protocol, thereby decoupling component logic from low-level protocol processing and ensuring the portability and modularity of component implementation.

[0040] Supports multiple inter-agent communication modes. In the peer-to-peer mode shown in Figure 3, each agent maintains an address table for all other peer agents. Communication occurs directly between agents without any intermediary components.

[0041] Figure 4 illustrates an alternative communication configuration of a distributed proxy system 400 according to some embodiments. In one embodiment, element 410 represents a name server residing in a node, which may be one of the nodes hosting one or more framework components, or another node. In name server mode, centralized name resolution service is provided by the name server. Proxy agents of the framework components query the name server to resolve the address of the destination proxy before initiating communication with the destination proxy. In another embodiment, element 410 represents a gateway server. In gateway mode, the centralized gateway server acts as an intermediary message distributor. Each proxy transmits data to the gateway server, which forwards the data to the destination proxy. In gateway mode, the proxy does not need to maintain knowledge of the addresses of other proxies.

[0042] Figure 5 is an illustration showing session management in a distributed proxy system according to one embodiment. In this embodiment, multiple proxy managers (e.g., 180a and 180b) run on multiple nodes and share the same model service 160 and / or database service 170. The model service 160 and / or database service 170 can run on the node where one of the proxy managers (180a or 180b) resides, or on another node. The model service 160 and database service 170 each use a corresponding session manager (560 and 570) to facilitate concurrent access from multiple clients, where the clients include the proxy managers (180a and 180b) and applications (550a and 550b). These session managers 560 and 570 are responsible for managing concurrent access while maintaining session-specific state for each client.

[0043] The model session used here begins when a client opens a session with model service 160 to access a specified model and ends when the client closes the session. Each client-to-model connection creates a unique model session. The model session is specific to the pairing of the model and the client. The database session begins when a client opens a session with database service 170 to access the database and ends when the client closes the session.

[0044] A model session management mechanism is provided to allow multiple clients in a distributed agent system to access model services while preserving system resource usage on edge devices. A multi-core device (e.g., a device with multiple neural processing units (NPUs)) can support concurrent sessions executed in parallel. However, requests within a model session are executed sequentially regardless of the number of NPUs available. When multiple concurrent model session requests come from the same model's service, the session manager 560 maintains a session state and session context for each model session. During a model session, the session manager 560 maintains a request history 561 and a session context 563 for each session. The request history 561 records pending requests from clients that are waiting to be sent to a given model in the model session. Furthermore, the session manager 560 maintains a global request queue 562 to track all pending requests from all clients to the model service in concurrent model sessions. The order of requests in the global request queue 562 can be first-in-first-out (FIFO), priority-based, or based on a predetermined strategy. Each entry in the global request queue 562 indicates a pending request from one of the clients, for one of the AI ​​models. As shown in Figure 6, the Session Manager 560 interleaves access to the model from multiple clients, enabling time-sharing access to the same model.

[0045] Session manager 560 retrieves requests from the request history 561 of all clients and queues them into a global request queue 562. The requests are then dequeued and dispatched to the appropriate model for execution. After a request is completed or the current execution of a given model is interrupted, session manager 560 dequeues the next request from the global request queue 562 for processing.

[0046] Before dispatching a client's request to the model, the session manager 560 performs two checks to determine whether the client's session context 563 needs to be updated. Once session context management is complete (i.e., the first and second checks), the new request is dispatched to the model.

[0047] In one embodiment, the session manager 560 performs an initial check to determine whether the client associated with the pending dispatch request (“new client”) is different from the client associated with the model’s last execution request (“last client”). If they are the same client, there is no need to update the session context 563. If the clients are different, the model’s last execution state is saved to the last client’s session context 563. This saved context replaces the session context 563 previously stored for that client.

[0048] If the first check indicates that the new client is different from the last client, the session manager 560 performs a second check to determine whether the new client has a previously saved session context 563 for the model. If such a context exists, it is loaded and used to restore the execution state of the model, allowing the session to continue as if the new client's session were continuous.

[0049] Figure 6 is a diagram illustrating the operation of a session manager 560 according to one embodiment. In this non-limiting example, request histories of four clients are shown: 561A, 561B, 561C, and 561D. Each request in the request history (e.g., Req1, Req2, etc.) was issued by the corresponding client and has not yet been executed. The session manager 560 schedules these requests into a global request queue 562 for execution according to a predetermined strategy. In an example of the global request queue 562, Cx represents a client, and Sx represents a session (where x = A, B, C, or D). MA represents model A, and MB represents model B. R1 and R2 represent Req1 and Req2, respectively. An arrow between two sessions indicates a session switch, resulting in a context switch. An arrow between two models indicates a model switch.

[0050] In one embodiment, the session manager 560 may preemptively interrupt model execution before the request is completed. In this case, the execution state may not be saved to the current client's session context 563. Depending on the operation policy, the client's last saved session context 563 may be cleared (e.g., invalidated or deleted).

[0051] Figure 7 is a flowchart illustrating a method 700 of a distributed agent system according to one embodiment. Non-limiting examples of distributed agent systems operable to perform method 700 may include any embodiments shown in Figures 1 through 5. Method 700 begins at step 710 when an agent manager receives a user request via one of the interactive peripherals in the distributed agent system. The distributed agent system includes multiple communicatively connected nodes. Each node is an edge device or a virtual machine (VM) running on an edge device. Interactive peripherals are coupled to a subset of nodes to receive user requests and output responses. At step 720, the agent manager sends a prompt to an artificial intelligence (AI) model managed by a model service based on the user request. At step 730, the agent manager receives an action plan from the AI ​​model. At step 740, the agent manager invokes at least one application based on the action plan to generate a response to the user request. The agent manager, AI model, model service, and at least one application reside on two or more nodes.

[0052] The following description shifts to the management of user session interruption and termination. A user session begins when a user sends a request to the agent manager to perform a task and ends when the task is completed. In some cases, a user session may be stopped before the requested task is completed. The stop request may be initiated by the user or a framework component.

[0053] Referring to Figure 1, a user session may be stopped by the user via a GUI, voice, etc., provided by interactive peripheral 190. This may occur when the user no longer wishes to wait for the ongoing user session to complete. Alternatively, application 150 (e.g., a high-privilege system service) may stop the user session. For example, application 150 may stop an ongoing user session when it detects an error or fault condition. When a hot condition (e.g., exceeding a hot threshold) is detected, a system service may take over the operation at the time of consumption. In one embodiment, an application 150 that is not currently used in an ongoing user session but has a higher priority than the task being performed in the ongoing user session may issue a request to stop the ongoing session. Depending on the system architecture, such a request may be directed to agent manager 180, model service 160, or both. In another embodiment of framework 105 that includes multiple agent managers, a first agent manager running on a first node may issue a stop request to stop a user session executing on a second node. The stop request is transmitted to a second agent manager on the second node via an inter-system communication channel. In the following description, the embodiments of Figure 1 are used as a non-limiting example of user session management. It should be understood that user session management can be performed in each embodiment of Figures 1 through 5. It should also be understood that user session management can be performed in a non-distributed agent system. In some embodiments, a separate device performing agent system operations may perform user session management.

[0054] Figure 8 is an illustration illustrating session phase management and context switching according to one embodiment. A user session may include multiple discrete processing phases, such as a speech-to-text (STT) phase 810, a model inference phase 820, an action phase 830, and a text-to-speech (TTS) phase 840.

[0055] In one embodiment, a stop request may be received during model inference phase 820. The stop request session may include a speech-to-text (STT) phase 831, an inference phase 832, and an action phase 833. The stop request interrupts inference phase 820, causing inference to pause 825. A context switch is performed when the inference phase 832 of the stop request begins. This occurs when the agent manager 180 uses the same model to handle both an ongoing user session and a new stop request session. In this case, ongoing model inference is interrupted to allow processing of the stop request. Referring to Figure 8, after processing the stop request, the node receiving the stop request may perform one of the following operations: (T1) Remove the ongoing user session, where the ongoing user session is abandoned and transitioned to a completed state (i.e., terminated); (T2) Start a new session, where the ongoing user session is discarded, and initialize and execute a new user session with a new inference phase 841 and a new session context (C2); (T3) Modify the user request, restore the session context (C0) saved at the start of the inference phase 820 of the ongoing user session, and continue the modified inference phase 831 based on the modified user request; (T4) Restore the ongoing user session, restore the session context (C1) from the point when the user session was interrupted, and continue the inference phase 821.

[0056] If a stop request is received at other stages of the user session (e.g., speech-to-text (STT) stage 810, action stage 830, or text-to-speech (TTS) stage 840), the agent manager 180 may interrupt and terminate the processing of that stage.

[0057] Figure 9 is a block diagram illustrating collaboration between multiple agent managers according to one embodiment. In one embodiment, multiple agent managers 180 running on respective devices may collaborate upon user request. In one embodiment, these devices are independent devices and do not share agent framework components. In one embodiment, these devices may include coordination functionality to enable their respective agent managers to collaborate upon user request. More specifically, the agent managers of these devices may collaborate through their respective interactive peripherals to fulfill user requests. Two non-limiting examples are provided below.

[0058] In the first example, user A and user B operate devices A and B, respectively. Each device includes an agent manager (180A, 180B), a set of applications (150A, 150B), interactive peripherals (190A, 190B), system functions (110A, 110B), model services (160A, 160B), and database services (170A, 170B). When user A sends a request to device A, such as “set up a meeting and mark the calendar for me and user B,” the agent manager 180A on device A may use an artificial intelligence (AI) model (e.g., one of the models managed by model service 160A) to generate multiple sub-requests to handle the semantic structure of user A’s request. For example, agent manager 180A may handle the request by breaking it down into two sub-request sequences. The first sub-request refers to itself to create a calendar entry for user A, including user B as a participant. The second sub-request is sent to the agent manager 180B on device B to create a corresponding calendar entry for user B, including user A as a participant.

[0059] In the second example, user A may have both device A and device B, both running agent managers 180A and 180B. Device A has a basic photo album app, while device B has a more powerful and resource-intensive photo enhancement app. When user A makes a request to device A, such as "get the latest photo and enhance it with device B, then send it back to me," agent manager 180A may process the request by breaking it down into a sequence of sub-requests. The first sub-request is a local sub-request to retrieve the latest photo from device A or the cloud and transfer it to device B. The second sub-request is a remote sub-request sent to agent manager 180B to apply the photo enhancement app to the photo and return the enhanced photo to device A. The third sub-request is a local sub-request to display the returned enhanced photo on device A.

[0060] It is understood that the example above, referring to Figure 9, also applies to a distributed proxy framework, where the framework components in device A and device B are distributed across multiple nodes. The term "proxy system" refers to a system in which a proxy framework is deployed. In one embodiment, the proxy system may be a single device. In another embodiment, the proxy system may be a distributed network of nodes.

[0061] Figure 10 is a flowchart illustrating a method 1000 for a collaborative agent manager according to one embodiment. While two agent managers (e.g., a first agent manager "Agent_A" and a second agent manager "Agent_B") are described in this example, it is understood that method 1000 extends to multiple (e.g., more than two) collaborative agent managers, for example, when the originating device (the source of the user request) uses multiple target devices to fulfill the user request. Referring to Figure 9, Agent_A and Agent_B may be examples of agent managers 180a and 180b. Furthermore, although "device" is used in the following examples, it is understood that method 1000 is applicable to scenarios where Agent_A and Agent_B are components of the respective agent system framework.

[0062] In one embodiment, method 1000 begins at step 1010 when Agent_A on the originating device receives a user request. At step 1020, Agent_A processes the user request using an artificial intelligence (AI) model to generate a sequence of sub-requests. The AI ​​model may semantically parse the user request while generating the sub-request sequence. The AI ​​model determines whether each sub-request is to be executed locally on the originating device or on the target device. The AI ​​model may also analyze dependencies in the sub-requests and determine the execution order of the sub-requests accordingly. At step 1030, Agent_A sends one or more sub-requests to Agent_B on the target device. In scenarios using multiple target devices, Agent_A may send the sub-requests to these target devices according to the execution order. In one embodiment, each sub-request may be accompanied by a session identifier or metadata to ensure proper handling of collaborative operations.

[0063] In step 1040, after receiving the sub-request from the first agent, Agent_B processes the sub-request using a second AI model on the target device. Agent_B processes the sub-request in a manner similar to processing a user request. Agent_B may use the second AI model to analyze the sub-request and plan an operation. Agent_B may invoke operations from available applications on the target device. In step 1050, Agent_B sends the application output from the target device to Agent_A. In step 1060, Agent_A generates a response to the user request on the original device based on the output.

[0064] In one embodiment, Agent_A can determine the target device for collaboration based on user actions, such as bumping, clicking, or bringing two devices close together. These actions can be detected by sensors in the original device as an indication of collaboration intent. For example, the two devices can detect each other's presence via Near Field Communication (NFC) or short-range wireless communication such as Bluetooth or Wi-Fi. Once Agent_B agrees to establish a connection with Agent_A, a temporary peer connection is established so that Agent_A can send sub-requests to Agent_B. Alternatively, Agent_A can determine the target device for collaboration based on the device identifier or other device identification information in the user request.

[0065] In some embodiments, the target device can receive sub-requests via any available wired or wireless channel, such as Bluetooth, Wi-Fi, Ethernet, cellular networks, etc. Sub-requests can be sent by Agent_A via notifications, such as text messages, emails, push notifications, SMS services, etc. Agent_B can receive and parse the notification to extract the sub-request content and process the sub-request. In one embodiment, the target device can provide a notification listening service, which can be implemented in an email or messaging application. If the user of the target device grants the necessary permissions, Agent_B can receive notifications from Agent_A and extract the sub-requests using the protocol and / or application programming interface (API) between the application and Agent_B. Agent_B can return a response, result, or acknowledgment to Agent_A.

[0066] Figure 11 is a block diagram illustrating an agent system 1100, whose operation is used to perform prompting session summarization, according to one embodiment. In agent system 1100, fulfilling a user request involves interaction between agent manager 180 and one or more AI models, one or more databases 1170, application 150, system function 110, and the user. The interaction between agent manager 180 and inference model 1169 (i.e., edge AI model) is referred to as a prompting session, in which agent manager 180 sends prompts to inference model 1169—containing information from the user, database 1170, application 150, agent manager 180 itself, and other models—and inference model 1169 generates an inference response and returns it to agent manager 180. Agent system 1100 is an edge-side system, and at least agent manager 180, inference model 1169, and summarization model 1168 reside on one or more edge devices.

[0067] The cue session may include multi-round cue-response exchanges between the agent manager 180 and the inference model 1169. When a new cue is sent to the inference model 1169, the model generates inference based on the combination of the new cue and all accumulated previous cue in the cue session. That is, the actual input to the current inference includes all cue accumulated in the cue session.

[0068] However, AI models typically impose token size limits on inference. The total token size includes the AI ​​model's inputs (i.e., cues) and outputs in a cue session. The token size limit restricts the size of accumulated cues in a cue session. As more cues accumulate, the number of tokens available for the model's output decreases. In some embodiments, the AI ​​model may impose a cue size limit that restricts the total length of the model's inputs in a cue session. Therefore, earlier cues are discarded when the accumulated cue size in a cue session exceeds the cue size limit, or when the total token size in a cue session exceeds the token size limit. Since earlier cues may contain information crucial to accurate inference, truncating them can lead to degraded or inaccurate results—an undesirable outcome in agent systems.

[0069] In one embodiment, the agent manager 180 may provide automatic prompt session summarization and history memory techniques to manage the cumulative prompt size in a prompt session. In one embodiment, the agent manager 180 automatically summarizes some or all prompts in a prompt session, generating a summary to retain key information while reducing length. For example, key information may include keywords, key sentences, key facts, etc., in each prompt. The summary may be a shorter version of the prompt, where long sentences are rephrased into equivalent shorter versions. This summary can then replace the original prompt in subsequent model inference, thereby preserving prompt history without exceeding symbol size limits.

[0070] In one embodiment, before sending a new prompt to the inference model 1169 in a prompting session, the agent manager 180 first sends the new prompt to the summarizing model 1168, which is also an edge AI model. The summarizing model 1168 generates a summary of the new prompt and sends the summary back to the agent manager 180. The agent manager 180 then uses the summary to prompt the inference model 1169 to continue the prompting session. Summarizing the new prompt is particularly helpful when the new prompt contains a large amount of information (e.g., a lengthy article).

[0071] Alternatively, before sending a new hint to the inference model 1169, the agent manager 180 can use the summary model 1168 to summarize the accumulated past hints in the hint session and use the summary to restart the new hint session. That is, the agent manager 180 uses the summary of accumulated past hints as the initial hint to start a new hint session for the inference model 1169.

[0072] In one embodiment, the summarization model 1168 can be a large language model (LLM) that summarizes the content by analyzing the input text, identifying key information, and then rephrasing it in a more concise form. This process can either extract key sentences directly from the text or generate new sentences that capture the main ideas.

[0073] In one embodiment, after the prompting session is completed, the agent manager 180 uses the inference model 1169, the summarization model 1168 or other AI model to summarize all the prompts of the session and stores the generated prompting session summary in the database 1170 for future reference.

[0074] The summarization process is used not only to manage cue size limits, but also to persist or remember key session information (such as user preferences, past actions, or application operation steps) in the database. This stored cue session summary can be used in future sessions to improve system performance, personalization, and / or efficiency.

[0075] Figure 12 is a flowchart illustrating a method 1200 for summarizing a prompt session, according to one embodiment. The method 1200 begins at step 1210, whereby an agent manager initiates a prompt session in response to a user request, wherein the prompt session includes multiple rounds of prompt-response exchanges between the agent manager and an inference AI model on an edge device. At step 1220, the agent manager sends a prompt to the summarizing AI model on the edge device to summarize the prompt. At step 1230, the agent manager receives a summary of the prompt from the summarizing AI model. At step 1240, the agent manager sends the summary to the inference AI model to generate an action plan. At step 1250, the agent manager invokes an application based on the action plan to generate a response to the user request.

[0076] In all the above-described embodiments of the proxy framework and proxy system, when the proxy manager coordinates the operations of applications, services, and / or devices (including services, devices, and similar components), it typically needs access to detailed application information (i.e., application metadata), including functionality, application programming interfaces (APIs), and internal data. This application information can be utilized by the proxy manager with the assistance of one or more artificial intelligence (AI) models to plan and trigger application operations. The application information can also be integrated into prompts sent to AI models or stored in one or more databases (including a retrieval-augmented generation (RAG) database) for the proxy manager to query.

[0077] Because the agent manager's operations depend on this application information, this information is typically stored in a database before the agent manager performs any operations on the application—for example, when the application is installed on an edge device or when the framework is initially deployed. This process is called deployment-time database setup.

[0078] However, there is a key limitation to the database settings at deployment time: application information may change over time. Therefore, the stored application information may be outdated when the agent manager coordinates application operations. For example, the database may contain menu data for a restaurant application (such as the KFC application), which is stored at deployment time, but the restaurant's menu may have changed when the agent manager interacts with the application, causing inaccurate or erroneous system behavior.

[0079] In one embodiment, the agent manager retrieves application information at runtime and stores it in a database for runtime querying. An application is referred to as a foreground application when it starts and is ready to perform an operation. Foreground applications also include those running in the background and brought to the foreground to perform operations. When an application becomes a foreground application or the agent manager itself starts, the agent manager queries the foreground application to retrieve the latest application information. When the agent manager receives new application information, it updates the database with the new application information.

[0080] To efficiently manage database updates, the system maintains a database update time for each application, which can be maintained by the application or the agent manager. The application's database update time indicates the time when the application's information was most recently updated in the database. In one embodiment, the application in the system maintains a corresponding information change time. The application's information change time indicates the time when its application information was most recently modified. If an application does not track its information change time, the current query time (relative to the time the agent manager sent the query) can be used as its information change time.

[0081] In the first embodiment, the database update time is maintained by each application. When an application receives a query from the agent manager, it compares its information change time with the stored database update time. If the information change time is newer than the database update time, the application returns the new application information to the agent manager and updates the database update time with the information change time. Then, the agent manager updates the database with the new application information. If the information change time is not newer than the database update time, the application notifies the agent manager that the application information has not changed since the last database update and does not need to be updated.

[0082] In the second embodiment, the database update time is maintained by the agent manager. When an application responds to a query from the agent manager, it returns its application information and its information change time. The agent manager then compares the received information change time with the application's stored database update time. If the information change time is newer than the database update time, the agent manager updates the database with the new application information and updates the database update time with the information change time. If the information change time is not newer than the database update time, the agent manager uses the application information in the database and ignores the application information from the application.

[0083] In one embodiment, when the database is updated with new application information, the agent manager can convert the data into embedded vectors and update the vector storage in the Retrieval Enhancement Generation (RAG) database to support efficient vector search operations.

[0084] Figure 13 is a flowchart illustrating an embodiment of a method 1300 for updating application information at runtime, executed by an agent manager on an edge device. The agent manager may be agent manager 180 in Figure 1 or any of the agent managers described above. The agent manager may be an agent manager in a distributed or non-distributed agent system. Method 1300 begins at step 1310, where the agent manager requests application information from the application at runtime. The application tracks the information change time, which indicates the time when the application information was last updated. At step 1320, the agent manager compares or obtains a comparison between the information change time and the database update time. The database update time indicates the time when the application information was last updated in a database on the edge device. At step 1330, the agent manager retrieves the application information from the application or the database based on which of the information change time and the database update time is more recent. At step 1340, the agent manager invokes the application to generate output. The application information includes one or more of the following: functionality, application programming interfaces (APIs), and internal application data.

[0085] Figure 14 is a block diagram illustrating one embodiment of device 1400 in the agent system. Device 1400 may be one of the nodes in a distributed agent framework or distributed agent system, as described in Figures 1 through 5. Device 1400 may also be a standalone device performing agent system operations. In some embodiments, device 1400 may be any device performing the agent manager operations described above, such as the embodiments shown in Figures 9 and 11.

[0086] Device 1400 includes processing hardware 1410, and further includes processor 1413 and AI hardware 1412. Non-limiting examples of processor 1413 include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor, a media processor, etc. Processor 1413 can perform operations of agent manager 180. Device 1400 also includes memory 1420, such as a static random access memory (SRAM) device, a dynamic random access memory (DRAM) device, a flash memory device, and / or other volatile or non-volatile memory devices. Memory 1420 can store machine-executable instructions for processor 1413 to perform operations of agent manager 180. In some embodiments, memory 1420 may also store agent framework components, such as system functions 110, applications 150, databases 172 and 173, user interfaces 191, 192, and / or 193, and / or edge model 164 (Figure 1).

[0087] Device 1400 may also include a network interface 1430, which may be a wired interface and / or a wireless interface. It should be understood that device 1400 is for illustrative purposes only; other hardware and software components are not shown.

[0088] Various functional elements or modules have been described herein. As will be understood by those skilled in the art, these functional modules are preferably implemented by circuits (whether dedicated or general-purpose circuits, which operate under the control of one or more processors and coded instructions), which typically include transistors configured to control the operation of the circuits according to the functions and operations described herein.

[0089] Although the present invention has been described with reference to several embodiments, those skilled in the art will recognize that the invention is not limited to the described embodiments and that modifications and changes can be made within the spirit and scope of the appended claims. Therefore, the description should be considered illustrative rather than restrictive. [Simplified Explanation of the Diagram]

[0011] This invention is illustrated by way of example rather than by way of limitation. In the accompanying drawings, similar elements are indicated by the same reference numerals. It should be noted that different references to "a" or "one" embodiment in this disclosure do not necessarily refer to the same embodiment; these references mean at least one. Furthermore, when a particular feature, structure, or characteristic is described in one embodiment, the submitter believes that those skilled in the art will be able to implement such features, structures, or characteristics in other embodiments, whether explicitly described or not. Figure 1 is a block diagram according to one embodiment, illustrating a proxy framework. Figure 2 is a block diagram according to one embodiment, illustrating the interaction between framework components. Figure 3 illustrates a peer-to-peer communication configuration of a distributed proxy system according to one embodiment. Figure 4 illustrates alternative communication configurations of a distributed proxy system according to some embodiments. Figure 5 is a diagram according to one embodiment, illustrating session management. Figure 6 is a diagram according to one embodiment, illustrating the operation of a session manager for managing model services. Figure 7 is a flowchart according to one embodiment, illustrating a method of a distributed proxy system. Figure 8 is a diagram according to one embodiment, illustrating session phase management and context switching. Figure 9 is a block diagram according to one embodiment, illustrating collaboration between multiple proxy managers. Figure 10 is a flowchart illustrating a method 1000 for a collaborative agent manager according to one embodiment. Figure 11 is a block diagram illustrating an agent system operating to perform a session summary prompt according to one embodiment. Figure 12 is a flowchart illustrating a method for prompting a session summary according to one embodiment. Figure 13 is a flowchart illustrating a method for runtime application information updates performed by an agent manager on an edge device according to one embodiment. Figure 14 is a block diagram illustrating a device in an agent system according to one embodiment.

Claims

1. A method for a collaborative agent manager, comprising: User requests are received by the first agent manager on the original device; The user request is processed using a first artificial intelligence (AI) model to generate a sequence of sub-requests; one or more sub-requests are sent to a second proxy manager on the target device; the second proxy manager processes one or more sub-requests using a second AI model on the target device; the application output on the target device is sent from the second proxy manager to the first proxy manager; and the first proxy manager generates a response to the user request on the original device based on the output.

2. The method as described in request item 1, wherein each sub-request is accompanied by a session identifier that identifies the user session initiated by the user request.

3. The method as described in request item 1, wherein the user request includes identification of the target device.

4. The method as described in claim 1, further comprising: The original equipment manufacturer (OEM) detects and identifies user actions near the target device.

5. The method as described in claim 1, further comprising: receiving a notification sent from the originating device by the target device; and extracting one or more sub-requests from the notification.

6. The method as described in request 1, further comprising: sending multiple sub-requests from the originating device to multiple agent managers on multiple target devices for processing.

7. The method as described in claim 1, further comprising: using a first AI model by the originating device to determine whether each sub-request is executed by the originating device or by the target device, and determining the execution order of the sub-requests based on dependencies in the sub-requests.

Images