A chip, an LLM-based agent system, and an agent device

Abstract

The application provides a chip, an LLM-based agent system and an agent device, wherein the chip comprises a task graph analysis and scheduling circuit, a micro-instruction decoding and execution circuit and an execution controller, and the task graph analysis and scheduling circuit and the micro-instruction decoding and execution circuit are connected with the execution controller, wherein: the task graph analysis and scheduling circuit is configured to analyze a received task graph, obtain a to-be-executed micro task and send the to-be-executed micro task to the micro-instruction decoding and execution circuit; the micro-instruction decoding and execution circuit is configured to convert the received micro task into a micro instruction and distribute the micro instruction to an idle execution core; and the execution controller is configured to coordinate the operation of each circuit in the chip. The above scheme integrates key functional modules such as task graph analysis and scheduling, micro-instruction decoding and execution and an execution controller at the hardware level, constructs a high-efficiency cooperative special hardware architecture and is beneficial to improving task execution efficiency and speed.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a chip, an LLM-based intelligent agent system, and an intelligent agent device. Background Technology

[0002] With the rapid development of artificial intelligence technology, intelligent systems centered on AI agents have demonstrated enormous potential in numerous fields such as automation, intelligent assistants, robot control, and enterprise services. AI agents possess autonomous capabilities in perception, thinking, planning, and action, and are considered a key force driving the development of a future intelligent society.

[0003] However, currently, the core decision-making logic and complex task scheduling of AI agents mostly rely on general-purpose processors or pure software implementations. When performing task planning, conditional judgment, and tool invocation, AI agents have to frequently invoke high-latency model inference and perform extensive data interactions, resulting in significant task execution delays and enormous consumption of computing resources. For scenarios with stringent real-time requirements, battery power, or large-scale deployment needs, execution efficiency has become a core bottleneck restricting the widespread application of AI agents. Summary of the Invention

[0004] The purpose of this application is to provide a chip, an LLM-based intelligent agent system, and an intelligent agent device to solve the above-mentioned problems.

[0005] In a first aspect, embodiments of this application provide a chip, the chip comprising: a task graph parsing and scheduling circuit, a microinstruction decoding and execution circuit, and an execution controller, wherein the task graph parsing and scheduling circuit and the microinstruction decoding and execution circuit are respectively connected to the execution controller, wherein:

[0006] The task graph parsing and scheduling circuit is configured to parse the received task graph, obtain the micro-tasks to be executed, and send the micro-tasks to be executed to the micro-instruction decoding and execution circuit.

[0007] The microinstruction decoding and execution circuit is configured to convert the received microtask into microinstructions and allocate the microinstructions to an idle execution core so that the execution core executes the microinstructions;

[0008] The execution controller is configured to coordinate the operation of the circuits in the chip to complete the microtask.

[0009] In the implementation of the above scheme, a highly efficient and collaborative dedicated hardware architecture was constructed by integrating key functional modules such as task graph parsing and scheduling, microinstruction decoding and execution, and execution controller at the hardware level. On the one hand, the parallel processing capability of the hardware circuit enables efficient task parsing, scheduling, and execution, which is beneficial to improving task execution efficiency and speed. On the other hand, compared with the high execution latency of pure software implementation, the above chip reduces execution latency through the close cooperation of various components, which is beneficial to improving the chip's responsiveness. Furthermore, hardware-level execution control improves the overall stability and reliability of the chip and reduces operational uncertainty caused by software failures. Moreover, while reducing execution latency, the above chip also reduces its dependence on general-purpose processor resources, which is beneficial to reducing the power consumption of the intelligent agent and provides a foundation for the widespread application of intelligent agents in scenarios with high real-time requirements, limited resources, or large-scale deployment.

[0010] In one implementation of the first aspect, the task graph parsing and scheduling circuit includes: a multi-parallel hardware scheduler, wherein:

[0011] The hardware scheduler, which operates in a multi-path parallel manner, is configured to perform parallel parsing and scheduling of the task graph.

[0012] In the implementation of the above scheme, a multi-path parallel hardware scheduler simultaneously parses multiple nodes in the task graph, quickly identifies each node and its dependencies, shortens the task decomposition time, and enables tasks to enter the execution phase more quickly, which is beneficial to improving the efficiency of task decomposition and scheduling. On the other hand, the multi-path parallel hardware scheduler can parse multiple task nodes at the same time, and reasonably allocate the micro-tasks to be executed to idle execution cores according to the dependencies and priorities between nodes. This parallel scheduling mechanism allows multiple execution cores of the chip to work simultaneously, which is beneficial to improving the parallel computing capability of the chip, thereby improving the speed and throughput of task execution.

[0013] In one implementation of the first aspect, the microinstruction decoding and execution circuit includes: a microinstruction decoder and a plurality of execution cores, wherein:

[0014] The micro-instruction decoder is configured to convert the received micro-task into the micro-instruction and send the micro-instruction to the corresponding execution core;

[0015] The execution core is configured to receive and execute the microinstructions.

[0016] In the implementation of the above scheme, the micro-instruction decoder can quickly convert complex micro-tasks into micro-instructions that can be directly executed by the underlying hardware. This process is efficient and accurate, laying the foundation for subsequent rapid execution. At the same time, multiple execution cores can process different micro-instructions in parallel, enabling the chip to perform multiple operations simultaneously, which is beneficial to improving the parallelism and speed of task execution. On the other hand, by setting multiple execution cores, the chip 100 can flexibly allocate resources and dynamically adjust the workload of each core according to the needs of the task, thereby better coping with the task requirements in different scenarios and improving the versatility and practicality of the chip.

[0017] In one implementation of the first aspect, the chip further includes: a multimodal fusion circuit connected to the execution controller, the multimodal fusion circuit comprising a multimodal data input interface, a data preprocessing unit, and a cross-modal fusion unit connected in sequence, wherein:

[0018] The multimodal data input interface is configured to transmit multimodal data to the data preprocessing unit;

[0019] The data preprocessing unit is configured to preprocess the multimodal data and input the preprocessed multimodal data into the cross-modal fusion unit.

[0020] The cross-modal fusion unit is configured to use a hardware-accelerated fusion algorithm to fuse the multimodal data and obtain the perception results.

[0021] In the implementation of the above scheme, the multimodal fusion circuit can provide the chip with richer environmental information by efficiently integrating multimodal data such as images, videos, audio, and text, enabling the intelligent agent to have a more comprehensive and in-depth understanding of complex scenes. On the other hand, the multimodal fusion circuit accelerates data processing and fusion at the hardware level, which is conducive to improving the multimodal data fusion time from data input to perception result output, and thus improving the real-time response capability of the chip. Furthermore, by completing data processing locally on the chip, the multimodal fusion circuit reduces the dependence on external transmission, thereby reducing the risk of data leakage and better protecting user privacy and data security.

[0022] In one implementation of the first aspect, the chip further includes: a condition judgment and branch execution circuit connected to the execution controller, wherein:

[0023] The condition judgment and branch execution circuit is configured to perform Boolean logic operations and generate branch decisions based on at least one of the following: chip internal state, the perception result, the return data obtained from the tool call interface, the external environment, and the knowledge enhancement result, so as to determine the task branch to be executed and the microtask sequence corresponding to the task branch in the task graph.

[0024] In the implementation of the above scheme, the condition judgment and branch execution circuit can perform Boolean logic operations in real time according to various input conditions to generate accurate branch decisions. This allows the chip to dynamically adjust task branches during task execution, adapt to complex and ever-changing environments, and enhance the chip's flexibility and adaptability. On the other hand, implementing condition judgment and branch execution at the hardware level helps improve the chip's decision-making speed and efficiency. Furthermore, integrating the condition judgment and branch execution circuit inside the chip reduces dependence on external components or software, which helps improve the chip's overall performance and reliability.

[0025] In one implementation of the first aspect, the chip further includes: an on-chip lightweight inference engine connected to the execution controller, wherein:

[0026] The on-chip lightweight inference engine is configured to convert received high-level task requests into the task graph.

[0027] In the implementation of the above solution, the on-chip lightweight inference engine directly implements the conversion process from high-level task requests to task graphs within the chip, saving latency and bandwidth consumption from repeated interactions with external large models. This reduces the startup time of agent tasks from seconds to milliseconds, which is beneficial to improving the overall response speed of the chip. On the other hand, since the conversion process can be completed locally, the original input can be parsed into a structured task graph without leaving the chip. Sensitive information always remains on-chip, which avoids the risk of leakage caused by network transmission and meets the compliance requirements of high privacy scenarios.

[0028] In one implementation of the first aspect, the chip further includes: a secure communication interface connected to the execution controller, wherein:

[0029] The secure communication interface is configured to de-identify high-level task requests through an encrypted channel, and send the de-identified high-level task requests to an external inference service to obtain the task graph generated by the external inference service based on the de-identified high-level task requests.

[0030] In the implementation of the above solution, de-identifying high-level task requests through a secure communication interface effectively protects user privacy and sensitive information, reduces the risk of data leakage during transmission, and meets the application requirements of high-privacy scenarios. On the other hand, using an encrypted channel to communicate with external inference services helps to ensure the security and integrity of data transmission, reduces the probability of data being stolen or tampered with during transmission, and thus improves the security of the chip. Furthermore, sending the de-identified high-level task requests to the external inference service can obtain more complex and accurate task graphs, which is beneficial to improving the accuracy and efficiency of task execution, while also expanding the functionality and application scenarios of the chip.

[0031] In one implementation of the first aspect, the chip further includes: a vector library retrieval circuit connected to the execution controller, the vector library retrieval circuit comprising an embedding generation unit and a vector retrieval unit connected in sequence, wherein:

[0032] The embedding generation unit is configured to embed the input information to obtain a high-dimensional vector;

[0033] The vector retrieval unit is configured to retrieve associated vectors from the local vector library based on the high-dimensional vector, and send the associated vectors to the execution controller.

[0034] In the implementation of the above scheme, the vector library retrieval circuit can quickly generate high-dimensional vector representations of input information and efficiently retrieve related knowledge in the local vector library, providing rich semantic information support for task execution. On the other hand, the vector library retrieval circuit enables the chip to complete knowledge retrieval and enhancement locally, avoiding the delay and leakage risks of external data transmission and ensuring data privacy and security. Furthermore, the hardware acceleration architecture of the vector library retrieval circuit achieves millisecond-level vector retrieval, supporting the chip to perform real-time knowledge enhancement in complex tasks, improving the accuracy and adaptability of task execution.

[0035] In one implementation of the first aspect, the embedding generation unit is configured as follows:

[0036] The high-dimensional vector is obtained by embedding the input information using a local lightweight embedding model.

[0037] In the implementation of the above scheme, using a local lightweight embedding model to generate high-dimensional vectors can reduce dependence on external services, reduce data transmission latency, and improve the response speed of the chip. On the other hand, input information does not need to leave the chip, thereby reducing the risk of data leakage. Furthermore, the local lightweight embedding model can run efficiently in resource-constrained environments, reducing the chip's power consumption and hardware requirements, making it suitable for more application scenarios and improving the chip's adaptability.

[0038] In one implementation of the first aspect, the embedding generation unit is configured as follows:

[0039] The input information is de-identified using a secure communication interface and then sent to an external inference service to obtain the high-dimensional vector output by the external inference service.

[0040] In the implementation of the above scheme, after the input information is anonymized through a secure communication interface, user privacy and sensitive data can be effectively protected, and the risk of data leakage during transmission can be reduced. On the other hand, by utilizing the powerful computing capabilities and advanced model architecture of external inference services, higher quality and more accurate high-dimensional vectors can be generated, improving the accuracy and depth of embedding generation. Furthermore, it enables the chip to flexibly expand its processing capabilities and call external resources on demand, which is conducive to improving the applicability of the above chip.

[0041] Secondly, embodiments of this application provide an LLM-based intelligent agent system, including: an external inference service and a chip provided as in the first aspect or any possible implementation of the first aspect, wherein the chip communicates with the external inference service through a secure communication interface.

[0042] Thirdly, embodiments of this application provide an intelligent agent device, including a chip as provided in the first aspect or any possible implementation thereof.

[0043] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing embodiments of this application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram of the chip structure provided in an embodiment of this application;

[0046] Figure 2 This is a schematic diagram illustrating the working principle of the multimodal fusion circuit provided in the embodiments of this application;

[0047] Figure 3 This is a schematic diagram illustrating the working principle of the vector library retrieval circuit provided in the embodiments of this application;

[0048] Figure 4 This is a schematic diagram of the structure of the chip provided in an embodiment of this application in a certain application scenario.

[0049] The numbers in the diagram are as follows:

[0050] 110. Task graph parsing and scheduling circuit; 120. Microinstruction decoding and execution circuit; 130. Execution controller; 140. On-chip lightweight inference engine; 150. Secure communication interface; 160. Multimodal fusion circuit; 170. Vector library retrieval circuit; 180. Condition judgment and branch execution circuit; 191. Agent task input / output interface; 192. Tool calling interface. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of this application, and are therefore merely examples and should not be used to limit the scope of protection of this application.

[0052] Currently, the core decision-making logic and complex task scheduling of AI agents mainly rely on general-purpose processors or pure software implementations. General-purpose processors are designed for versatility rather than the specific needs of intelligent agent tasks, resulting in their inability to fully utilize hardware resources when executing AI agent tasks and hindering efficient parallel task processing. On the other hand, pure software-implemented AI agent systems are susceptible to factors such as operating system scheduling, concurrency control, and software bugs, leading to a lack of determinism in operation and making them unsuitable for application scenarios with high real-time and reliability requirements.

[0053] Furthermore, during task planning, conditional judgment, and tool invocation, the AI ​​Agent frequently needs to invoke model inference. Current large language models (LLMs) typically have a massive parameter scale and complex structure, requiring significant computational resources for a single inference operation, making it difficult to meet the demands of real-time tasks. Simultaneously, these inference processes often require interaction with cloud servers via the network, and data transmission and network latency further exacerbate execution delays. During task execution, the AI ​​Agent needs to adjust its behavior path and make decisions in real time based on environmental perception results and user feedback. However, current software implementations often consume considerable computational resources and time for conditional judgments and branch execution, making it difficult to complete complex logical judgments with extremely low latency. This results in a slow response speed for the AI ​​Agent to environmental changes, hindering timely and accurate decision-making and impacting overall execution efficiency.

[0054] In view of this, this application provides a chip that integrates key functional modules such as task graph parsing and scheduling, microinstruction decoding and execution, and execution controller at the hardware level, constructing a highly efficient and collaborative dedicated hardware architecture. On the one hand, through the parallel processing capabilities of the hardware circuits, efficient task parsing, scheduling, and execution are achieved, which is beneficial to improving task execution efficiency and speed. On the other hand, compared with the high execution latency of pure software implementation, the above-mentioned chip reduces execution latency through the close cooperation of various components, which is beneficial to improving the chip's responsiveness. Furthermore, hardware-level execution control improves the overall stability and reliability of the chip and reduces operational uncertainty caused by software failures. Moreover, while reducing execution latency, the above-mentioned chip reduces its dependence on general-purpose processor resources, which is beneficial to reducing the power consumption of intelligent agents and provides a foundation for the widespread application of intelligent agents in scenarios with high real-time requirements, limited resources, or large-scale deployment.

[0055] Before providing a detailed explanation of the embodiments of this application, the concepts involved in the embodiments of this application will be explained first.

[0056] AI Agent: A highly intelligent agent. AI Agent can not only understand user instructions and intentions, but also actively formulate plans and execute a series of complex operations to achieve specific goals based on the environment and task requirements. The core capabilities of AI Agent include: (1) Intelligent decision-making: AI Agent can perform multi-round reasoning analysis based on set goals and environmental information, autonomously formulate reasonable task execution paths, and has the ability to dynamically adjust. (2) Task execution: AI Agent has the ability to act independently and can automatically call various tools, API interfaces or external system resources to efficiently execute multi-step complex tasks. (3) Observation results: During task execution, AI Agent can continuously monitor environmental changes and feedback information, judge whether each step has achieved the expected goal, and respond in a timely manner. (4) Memory storage: Through short-term and long-term memory mechanisms, AI Agent can record task progress, interaction history and context data to support continuous behavior and strategy optimization.

[0057] A task graph is a structure for representing an agent's task using a Directed Acyclic Graph (DAG). It consists of nodes and edges: each node represents a specific action or subtask to be performed, such as "reading sensor data," "calling a weather API," "determining if it's raining," or "sending an alert." Each directed edge represents the dependency or execution order between nodes, indicating which action must be completed first and which actions can be performed in parallel. Because loops are not allowed in the graph, the entire task flow always proceeds in a unidirectional manner, facilitating parallel scheduling and ensuring predictable results. During runtime, the agent first parses natural language instructions into this structured task graph, and then dedicated hardware units on the chip schedule and execute tasks concurrently according to the graph, thereby reducing latency, saving energy, and improving determinism.

[0058] Microtasks typically refer to the smallest unit of task that the operating system or hardware scheduler needs to process; they are a subdivision of tasks at the software level. In the context of chips, they may involve hardware multithreading, task queue management, or event-driven processing. Microtasks are more lightweight than traditional threads / processes.

[0059] Microinstructions: Microinstructions are the smallest hardware operation units when executing instructions, belonging to the concept of microarchitecture. A machine instruction can be decomposed into multiple microinstructions (such as fetch, decode, execute, and write back), which have a finer granularity than machine instructions and can directly control the data path (such as opening registers and triggering ALU operations).

[0060] The application scenarios involved in the embodiments of this application are described below:

[0061] The aforementioned chip can be used in intelligent agent task execution scenarios with requirements for high efficiency, security, and low latency, such as:

[0062] (1) General autonomous intelligent agent systems, for example:

[0063] Personal intelligent assistant: A locally running AI assistant (such as a voice assistant on a mobile phone / PC) that can quickly respond to the user's personalized needs;

[0064] Offline robots: such as home service robots and educational robots, rely on chips to complete environmental perception (vision / voice), task planning (such as "cleaning the kitchen") and tool recall (controlling robot vacuums and robotic arms, etc.) in environments without network access.

[0065] (2) Multi-agent cooperative systems, for example:

[0066] Industrial automation scenarios: For example, in multi-robot collaboration scenarios in factories (such as logistics handling and quality inspection), the above-mentioned chip ensures low-latency communication between intelligent agents through hardware-level task scheduling.

[0067] Smart city scenarios: For example, in a traffic management system, multiple intelligent agents (such as traffic light controllers and autonomous vehicles) share anonymized traffic data (such as "congestion level" rather than license plate information) locally through the aforementioned chip to collaboratively optimize traffic light timing or route planning.

[0068] (3) Enterprise intelligent agent automation scenarios, for example:

[0069] Financial risk control systems: For example, intelligent agent systems in banks or securities companies analyze local transaction data in real time (such as user behavior and transaction images) and automatically trigger high-frequency risk control strategies (such as freezing suspicious transactions).

[0070] Medical diagnostic assistance systems, such as hospital AI diagnostic agents, process patient images (such as X-rays) and medical record text locally on the aforementioned chip. They quickly generate preliminary diagnostic suggestions through multimodal fusion and only upload the desensitized abstract conclusions (such as "suspected pneumonia") to the cloud for LLM review.

[0071] (4) Edge computing and IoT scenarios, for example:

[0072] Intelligent security scenarios: For example, by deploying the above-mentioned chip in a camera, the chip can analyze the video stream in real time and detect abnormal behavior (such as intruders) locally, thereby triggering an alarm when abnormal behavior occurs.

[0073] Autonomous driving scenarios: For example, the onboard intelligent agent uses the aforementioned chip to fuse multimodal data such as LiDAR and cameras, and plans the path locally (such as emergency avoidance), only requesting cloud LLM support in complex scenarios (such as rare obstacle recognition).

[0074] The chip provided in the embodiments of this application will now be described in detail.

[0075] Example 1:

[0076] Please see Figure 1 This application provides a chip 100, which includes: a task graph parsing and scheduling circuit 110, a microinstruction decoding and execution circuit 120, and an execution controller 130. The task graph parsing and scheduling circuit 110 and the microinstruction decoding and execution circuit 120 are respectively connected to the execution controller 130, wherein:

[0077] The task graph parsing and scheduling circuit 110 is configured to parse the received task graph, obtain the micro-tasks to be executed, and send the micro-tasks to be executed to the micro-instruction decoding and execution circuit.

[0078] The microinstruction decoding and execution circuit 120 is configured to convert received microtasks into microinstructions and allocate the microinstructions to idle execution cores so that the execution cores can execute the microinstructions;

[0079] The execution controller 130 is configured to coordinate the operation of the various circuits in the chip to complete the microtask.

[0080] The working principle of the chip 100 to complete micro-tasks is as follows:

[0081] When the aforementioned chip 100 receives a complex task (such as a high-level task request input by a user), it first transforms the task into a structured representation (for specific transformation schemes, please refer to the subsequent description of the on-chip inference engine and external inference services), namely, a task graph. The main function of the task graph parsing and scheduling circuit 110 is to parse the received task graph, decomposing the complex task into a series of micro-tasks to be executed. These micro-tasks can be regarded as the basic units constituting the entire task, and each micro-task represents a specific and relatively simple operation step. After parsing, the task graph parsing and scheduling circuit 110 sends these micro-tasks to be executed to the microinstruction decoding and execution circuit.

[0082] After receiving a microtask, the microinstruction decoding and execution circuit 120 converts each microtask into specific microinstructions that the underlying layer of the aforementioned chip 100 can recognize and execute. Microinstructions are low-level commands that the chip hardware can directly execute; each microtask corresponds to a series of specific microinstructions. These microinstructions are then allocated to currently idle execution cores. Execution cores are the hardware units within the chip that actually perform operations; they can complete corresponding calculations or operations based on the received microinstructions. The microinstruction decoding and execution circuit 120 rationally allocates microinstructions to idle execution cores based on their workload, thereby ensuring the chip's parallel processing capability and efficient execution.

[0083] The execution controller 130 acts as a coordinator throughout the process. It monitors the operational status of the task graph parsing and scheduling circuit 110 and the microinstruction decoding and execution circuit 120 in real time, ensuring that microtasks are processed and executed in the correct order and logical relationships. When the task graph parsing and scheduling circuit 110 is parsing the task graph, the execution controller 130 provides necessary control signals and timing arrangements for the parsing and scheduling operations based on the structure of the task graph and the priority of the tasks. Similarly, during the process of the microinstruction decoding and execution circuit 120 converting microtasks into microinstructions and allocating execution cores, the execution controller 130 coordinates the work of each execution core, ensuring they can work together efficiently and avoid resource conflicts and operational chaos. Furthermore, the execution controller 130 can also handle abnormal situations during task execution, such as microtask execution failures or execution core malfunctions, ensuring the smooth completion of the entire task by reallocating tasks or activating backup execution cores.

[0084] In summary, the task graph parsing and scheduling circuit 110, the microinstruction decoding and execution circuit 120, and the execution controller 130 cooperate with each other, enabling the aforementioned chip 100 to efficiently and orderly complete complex micro-tasks.

[0085] The hardware implementation of the task graph parsing and scheduling circuit 110, the microinstruction decoding and execution circuit 120, and the execution controller 130 is described below:

[0086] Optionally, the task graph parsing and scheduling circuit 110 mentioned above includes: a multi-parallel hardware scheduler, wherein:

[0087] A multi-path parallel hardware scheduler is configured to perform parallel parsing and scheduling of the task graph.

[0088] The aforementioned hardware scheduler is a circuit structure primarily used for parallel parsing and scheduling of task graphs. It can process multiple task nodes simultaneously, identify dependencies between tasks, and allocate tasks reasonably to the execution cores of the chip for processing. The design goal of the hardware scheduler is to maximize the parallel processing capability of the chip, reduce task execution latency, and improve overall efficiency. The main functions of the hardware scheduler include: (1) Parallel parsing: The hardware scheduler can simultaneously parse multiple nodes in the task graph and quickly identify tasks that can be executed in parallel. This parallel parsing capability greatly shortens the task decomposition time and improves the efficiency of task scheduling. (2) Dependency identification: During the parsing of the task graph, the hardware scheduler will identify the dependencies between task nodes. For example, some tasks must be completed before they can start execution. The scheduler will reasonably arrange the execution order of tasks based on these dependencies. (3) Task allocation: The hardware scheduler will allocate tasks to idle execution cores based on task priority and the availability of execution cores. It will dynamically adjust the task allocation strategy to ensure efficient utilization of chip resources. (4) Real-time scheduling: The hardware scheduler can monitor the execution status of tasks in real time and dynamically adjust the priority and scheduling strategy of tasks according to the progress of tasks and resource usage. This real-time scheduling capability enables the chip to respond quickly to task changes and ensure efficient task execution.

[0089] For example, the aforementioned hardware scheduler can be a dedicated scheduler or a general-purpose scheduler. A dedicated scheduler is a special-purpose circuit designed for a specific purpose, and its processing flow is generally immutable. Multiple hardware schedulers can be combined to achieve different functions. For instance, a dedicated scheduler may include an application-specific integrated circuit (ASIC). Dedicated schedulers have a relatively fixed processing flow and high processing efficiency. A general-purpose scheduler is a circuit structure whose processing flow can be changed through software programming. A general-purpose processor may include a processor with a software-programmable microprocessor core. The processing flow of a general-purpose scheduler is more flexible, and the aforementioned chip 100 can be adapted to more application scenarios through software programming.

[0090] Compared to the serial processing method used in traditional task scheduling, the above scheme uses a multi-path parallel hardware scheduler to simultaneously parse multiple nodes in the task graph, quickly identify each node and its dependencies, shorten the task decomposition time, and enable tasks to enter the execution phase more quickly, which is beneficial to improving the efficiency of task decomposition and scheduling. On the other hand, the multi-path parallel hardware scheduler can parse multiple task nodes at the same time and reasonably allocate the micro-tasks to be executed to idle execution cores according to the dependencies and priorities between nodes. This parallel scheduling mechanism allows multiple execution cores of the chip 100 to work simultaneously, which is beneficial to improving the parallel computing capability of the chip 100, thereby improving the speed and throughput of task execution.

[0091] Optionally, the microinstruction decoding and execution circuit 120 mentioned above includes: a microinstruction decoder and multiple execution cores, wherein:

[0092] The micro-instruction decoder is configured to convert received microtasks into micro-instructions and send the micro-instructions to the corresponding execution core;

[0093] The execution core is configured to receive and execute microinstructions.

[0094] The aforementioned micro-instruction decoder is a specialized hardware circuit primarily responsible for handling the conversion from microtasks to micro-instructions. Specifically, after receiving microtasks from the task graph parsing and scheduling circuit 110, the micro-instruction decoder translates each microtask into a set of micro-instructions. These micro-instructions are low-level operation commands that the chip's underlying hardware can directly understand and execute. The micro-instruction decoder searches for the corresponding micro-instruction sequence in its internally stored instruction set mapping table based on the type of the microtask. At the hardware level, the micro-instruction decoder can consist of a control memory, an instruction register, and decoding logic. The control memory stores all possible micro-instruction sequences, the instruction register temporarily holds the microtask to be translated, and the decoding logic is responsible for converting the microtask into the corresponding micro-instructions. By decomposing complex operations into a series of simple micro-instructions, the micro-instruction decoder enables the aforementioned chip 100 to execute various tasks more efficiently and flexibly. Simultaneously, it simplifies hardware design and reduces the complexity of the control logic.

[0095] The aforementioned execution core is primarily responsible for executing microinstructions generated by the microinstruction decoder. Specifically, after receiving a microinstruction, the execution core executes the corresponding operation based on the opcode within the microinstruction. These operations may include arithmetic and logical operations (such as addition, subtraction, AND, OR, etc.), data transfer (such as from register to register, from register to memory, etc.), control flow operations (such as conditional jumps, loops, etc.), and interaction with other hardware modules (such as calling external tools, accessing vector libraries, etc.). The execution core may include an Arithmetic Logic Unit (ALU), a register set, a data path, and a control unit. The ALU is responsible for performing arithmetic and logical operations, the register set is used to temporarily store data and intermediate results, the data path is responsible for data transmission and flow, and the control unit generates control signals based on the opcode of the microinstruction to coordinate the work of various components. The hardware implementation of the execution core makes the execution of microinstructions efficient and fast, fully utilizing the chip's parallel processing capabilities. Through the collaborative work of multiple execution cores, the chip can process multiple microinstructions simultaneously, thereby improving task execution efficiency and throughput.

[0096] The collaborative work of the micro-instruction decoder and the execution core enables the aforementioned chip 100 to efficiently convert complex micro-tasks into micro-instructions executable by the chip's underlying hardware, and to quickly complete task execution through hardware acceleration.

[0097] The micro-instruction decoder in the above scheme can quickly convert complex micro-tasks into micro-instructions that can be directly executed by the underlying hardware. This process is efficient and accurate, laying the foundation for subsequent rapid execution. At the same time, multiple execution cores can process different micro-instructions in parallel, enabling the chip 100 to perform multiple operations simultaneously, which is beneficial to improving the parallelism and speed of task execution. On the other hand, by setting multiple execution cores, the chip 100 can flexibly allocate resources and dynamically adjust the workload of each core according to the needs of the task, thereby better coping with the task requirements in different scenarios and improving the versatility and practicality of the chip 100.

[0098] For example, the execution controller 130 described above mainly achieves precise control of the task execution process through hardware circuitry, ensuring that the various modules work together to complete complex intelligent agent tasks. The execution controller 130 can be implemented in at least one of the following two ways:

[0099] The first implementation method: using a finite state machine (FSM);

[0100] In this implementation, hardware-level state registers and combinational logic can be used to construct a state transition graph. Each state represents a sub-state of the agent's task, such as "waiting for input," "calling an API," "determining a branch," or "writing a result." The state register stores the current state, while the combinational logic calculates the next state and triggers the corresponding action based on the current state and the input signal. This approach features simple hardware implementation, clear control logic, and fast execution speed, making it particularly suitable for scenarios with relatively fixed task flows and a limited number of states.

[0101] The second implementation method is to adopt the RISC-V (Reduced Instruction Set Computer V) implementation method.

[0102] RISC-V is an open-source instruction set architecture. It employs the principles of reduced instruction set computing, featuring a concise, efficient, and modular design that supports various data widths (e.g., 32-bit, 64-bit, 128-bit). In this approach, a small RISC-V core can be used to execute the backbone microinstructions of the controller 130. The RISC-V core runs the control program, managing the execution flow of tasks, including task initiation, pausing, resumption, and termination. For computationally intensive modules such as MCP (Multimodal Computing and Perception) inference and tool calls, the RISC-V core can distribute these tasks to dedicated hardware accelerators or coprocessors. This approach provides greater flexibility and programmability, adapting to complex and ever-changing task requirements, while also allowing for improved execution efficiency for specific tasks through hardware accelerators.

[0103] The third implementation method: The execution controller 130 uses a finite state machine by default to implement deterministic scheduling. When a task that requires dynamic programming is detected, it automatically switches to RISC-V microcontroller mode. This method can improve the flexibility and determinism of the above-mentioned chip.

[0104] The following describes the scheme for obtaining the task graph using the aforementioned chip 100. The task graph in the above scheme can be implemented through at least one or more of the following three methods:

[0105] The first implementation method: using an on-chip lightweight inference engine;

[0106] Optionally, the aforementioned chip 100 further includes: an on-chip lightweight inference engine 140 connected to the execution controller 130, wherein:

[0107] The on-chip lightweight inference engine 140 is configured to convert received high-level task requests into task graphs.

[0108] The aforementioned on-chip lightweight inference engine 140 is a hardware module designed for efficient operation on a chip. Its core function is to convert the received high-level task requests into a structured task graph (DAG). The on-chip lightweight inference engine 140 can be a large language model deployed in the processor, or it can be a large language model whose entire functions (including word segmentation, embedding, attention calculation, feedforward network and decoding sampling, etc.) are customized into digital logic and implemented on-chip memory array. The hardware-fixed large language model has the following characteristics: (1) Network weight fixation: All parameters required for inference (including embedding matrix, weights and output layer, etc.) are burned into the on-chip SRAM at one time. The read and write latency is less than 5ns, and there is no need for off-chip DRAM transportation, which saves power and prevents snooping. (2) Computation array: A hybrid architecture of systolic array and SIMD (Single Instruction, Multiple Data) vector units can be adopted. In this architecture, the systolic array completes the matrix multiplication in multi-head attention, and the SIMD vector units perform layer normalization, activation function and element-wise operation in the feedforward network in parallel.

[0109] The entire inference process of the on-chip lightweight inference engine 140 is controlled by a finite state machine. The inference process mainly includes: receiving high-level task request text; triggering sub-modules such as embedding, attention, feedforward network FFN, residual, and normalization layer by layer; and finally outputting a token through sampling logic, while generating the corresponding structured DAG (task graph) and sending it to the task graph parsing and scheduling circuit 110.

[0110] The working principle of the above-mentioned task graph parsing and scheduling circuit 110, microinstruction decoding and execution circuit 120, execution controller 130, and on-chip lightweight inference engine 140 to achieve on-chip inference is as follows:

[0111] The on-chip lightweight inference engine 140 receives a high-level task request from the user (the high-level task request can be natural language or a mixed voice / text instruction). The on-chip lightweight inference engine 140 performs a lightweight large model inference inside the chip, directly converting the abstract requirements into a structured task graph (DAG). Each node in the graph represents an independently executable microtask, and the edges represent dependencies or sequence relationships. Then, the task graph is sent to the task graph parsing and scheduling circuit 110.

[0112] After receiving the task graph, the task graph parsing and scheduling circuit 110 scans all nodes in parallel to identify which tasks can be performed in parallel and which must be performed sequentially. Then, it sends the microtasks to the microinstruction decoding and execution circuit 120 according to their priority and resource usage.

[0113] After receiving a microtask, the microinstruction decoding and execution circuit 120 translates each microtask into a lower-level microinstruction and assigns these microinstructions to the currently idle execution core. The execution core then performs specific operations such as calculation, memory access, and peripheral device calls.

[0114] The execution controller 130 constantly monitors the entire process: on the one hand, it adjusts the priority of the task queue based on the real-time feedback from the task graph parsing and scheduling circuit 110; on the other hand, it determines whether to continue, skip, or retry a micro-task based on the execution status returned by the micro-instruction decoding and execution circuit 120. When condition judgment or external tool calls are required, the execution controller 130 can also trigger the corresponding unit in a timely manner and summarize the results to ensure that the entire task chain moves forward without interruption.

[0115] In summary, the aforementioned chip 100 completes the closed loop of "high-level task instruction → task graph → micro-task → micro-instruction → execution result" locally, without sending raw data or sensitive information out of the chip, thus ensuring low latency and high throughput while protecting user privacy.

[0116] The on-chip lightweight inference engine 140 in the above solution directly implements the conversion process from high-level task requests to task graphs within the chip, saving latency and bandwidth consumption from repeated interactions with external large models. This reduces the startup time of agent tasks from seconds to milliseconds, which helps improve the overall response speed of the chip 100. On the other hand, since the conversion process can be completed locally, the original input can be parsed into a structured task graph without leaving the chip. Sensitive information always remains on-chip, which avoids the risk of leakage caused by network transmission and meets the compliance requirements of high privacy scenarios.

[0117] The second implementation method: using an external inference service;

[0118] It should be noted that currently, many intelligent agent systems relying on Large Language Models (LLMs) still require uploading massive amounts of raw or sensitive data (including user personal information, internal enterprise documents, real-time audio and video streams, etc.) to the cloud for centralized processing. Data can be intercepted, misused, or leaked at every stage of transmission and cloud computing, and this vulnerability has become the primary obstacle to the large-scale deployment of intelligent agent technology. How to ensure data privacy is not violated while utilizing the powerful inference capabilities of Large Language Models (LLMs) has become a core bottleneck that urgently needs to be overcome. Therefore, this application provides the following solution in its embodiments:

[0119] Optionally, the chip 100 further includes a secure communication interface 150 connected to the execution controller 130, wherein:

[0120] The secure communication interface 150 is configured to de-identify high-level task requests through an encrypted channel and send the de-identified high-level task requests to an external inference service to obtain a task graph generated by the external inference service based on the de-identified high-level task requests.

[0121] The aforementioned external inference service is a cloud-based or local server-level computing resource deployed outside of chip 100 and connected to chip 100 via a secure communication interface 150. Its main responsibilities include: running large language models or hybrid inference systems based on anonymized high-level task requests, and returning a Directed Acyclic Graph (DAG) that can be directly parsed and executed by the chip. In essence, the external inference service acts as the "cloud brain" of the chip, providing local intelligent agents with deep thinking and task planning capabilities beyond on-chip computing power while maintaining minimal data exposure.

[0122] The aforementioned secure communication interface 150 is a hardware module within the aforementioned chip 100, responsible for secure communication with external inference services. It de-identifies high-level task requests via an encrypted channel and sends the processed requests to the external inference service to obtain the task graph.

[0123] The secure communication interface 150 may include the following hardware:

[0124] (1) Encryption module: This includes a hardware security module, which can be used to execute encryption algorithms such as Advanced Encryption Standard (AES) and RSA. The hardware security module can be a separate chip or a secure area on a chip, used to store encryption keys and perform encryption operations.

[0125] (2) Desensitization Engine: A dedicated hardware accelerator can be used to identify and process sensitive data (such as personal identification information, corporate secrets, etc.) and desensitize it before sending. The desensitization engine can identify sensitive data based on predefined rules or pattern matching.

[0126] (3) Physical layer communication module (i.e., encrypted channel): Supports multiple encrypted communication protocols, the specific implementation of which depends on the chip's design goals and application scenarios. These protocols are implemented through hardware accelerators, thereby improving communication speed and security.

[0127] The working principle of the above-mentioned task graph parsing and scheduling circuit 110, microinstruction decoding and execution circuit 120, execution controller 130, and chip-secure communication interface 150 for external inference is as follows:

[0128] When the aforementioned chip 100 needs to obtain a task graph through an external inference service, the execution controller 130 first receives a high-level task request. Then, the execution controller 130 sends the received high-level task request to the secure communication interface 150. The secure communication interface 150 desensitizes the high-level task request, removing sensitive information, and sends the desensitized request to the external inference service through an encrypted channel. Upon receiving the request, the external inference service generates the corresponding task graph and returns it to the secure communication interface 150. The secure communication interface 150 decrypts the received task graph and passes it to the execution controller 130. The execution controller 130 sends the task graph to the task graph parsing and scheduling circuit 110. The task graph parsing and scheduling circuit 110 parses the task graph, identifies each microtask and its dependencies, and sends the microtasks to the microinstruction decoding and execution circuit 120. The microinstruction decoding and execution circuit 120 converts the microtasks into microinstructions and allocates them to the execution core for execution. During execution, the execution controller 130 coordinates the operation of each circuit to ensure the smooth execution of the task. In this way, the aforementioned chip 100 can utilize external inference services to obtain task graphs and execute tasks efficiently locally, while ensuring data security and privacy.

[0129] The above solution uses the secure communication interface 150 to anonymize high-level task requests, effectively protecting user privacy and sensitive information, reducing the risk of data leakage during transmission, and meeting the application requirements of high privacy scenarios. On the other hand, using an encrypted channel to communicate with external inference services helps to ensure the security and integrity of data transmission, reducing the probability of data being stolen or tampered with during transmission, thereby improving the security of the chip 100. Furthermore, sending the anonymized high-level task requests to the external inference service can obtain more complex and accurate task graphs, which is beneficial to improving the accuracy and efficiency of task execution, while expanding the functionality and application scenarios of the chip 100.

[0130] The third implementation method: The on-chip lightweight inference engine 140 and the external inference service work together to obtain the task graph;

[0131] In this embodiment, chip 100 can choose to perform either on-chip inference or external inference, or it can adopt a cascaded approach of first performing on-chip inference and then external inference, wherein:

[0132] The first implementation method involves choosing between on-chip inference and external inference. An example of this implementation is:

[0133] The execution controller 130 first estimates the complexity of the high-level task request. If the task keywords, historical statistics, or resource consumption thresholds are all within the local capability range (e.g., including common single-turn question-and-answer sessions and simple tool calls), then the on-chip lightweight inference engine 140 is used directly for inference. The on-chip lightweight inference engine 140 returns the task graph in milliseconds and immediately sends it to the task graph parsing and scheduling circuit 110.

[0134] When the execution controller 130 determines that the task exceeds the capabilities of the local model (such as excessively long context, rare domain terms, probability thresholds below a preset threshold, etc.), it directly sends the high-level task request to the secure communication interface 150. The secure communication interface 150 performs desensitization and encryption before sending it to the external inference service, and then decrypts the task graph returned by the external inference service and forwards it to the task graph parsing and scheduling circuit 110.

[0135] The second implementation method involves performing on-chip inference first, followed by external inference. An example of this implementation is:

[0136] The execution controller 130 first calls the on-chip lightweight inference engine 140 to generate a coarse-grained task graph. Then, the execution controller 130 scans the key nodes of the coarse-grained task graph. If a node is found to be marked as requiring deep inference or with a confidence level below a threshold, the sub-request corresponding to that node is extracted and sent to an external inference service via the secure communication interface 150 to obtain a more refined sub-task graph. Then, the original coarse-grained task graph is merged with the more refined sub-task graph to form the final executable task graph.

[0137] Example 2:

[0138] It should be noted that intelligent agents in related technologies still face significant technical bottlenecks in rapidly fusing and effectively utilizing multi-modal information such as images, speech, and text, as well as in extracting required knowledge from massive knowledge bases in real time and with high accuracy. This directly limits their ability to comprehensively perceive and make in-depth decisions regarding complex environments. Therefore, this application provides the following solution:

[0139] Optionally, based on the above embodiment 1, the chip 100 may further include: a multimodal fusion circuit 160 connected to the execution controller 130, the multimodal fusion circuit 160 including a multimodal data input interface 161, a data preprocessing unit 162 and a cross-modal fusion unit 163 connected in sequence, wherein:

[0140] The multimodal data input interface 161 is configured to transmit multimodal data to the data preprocessing unit;

[0141] The data preprocessing unit 162 is configured to preprocess the multimodal data and input the preprocessed multimodal data into the cross-modal fusion unit;

[0142] The cross-modal fusion unit 163 is configured to use a hardware-accelerated fusion algorithm to fuse multimodal data and obtain perception results.

[0143] The aforementioned multimodal fusion circuit 160 is a hardware module in chip 100, used to process and fuse multimodal data, including images, video, audio, and text. The functions and hardware implementation of the aforementioned multimodal data input interface 161, data preprocessing unit 162, and cross-modal fusion unit 163 are described below:

[0144] (1) Multimodal data input interface 161;

[0145] Main function: Transmit multimodal data to data preprocessing unit 162;

[0146] Hardware implementation methods include: setting up various dedicated high-speed interfaces, such as CSI (Camera Interface) and MIPI (Mobile Industry Processor Interface) for image and video data, I2S (Integrated Circuit Voice Interface) for audio data, and UART / SPI for text data. These interfaces can directly receive raw data from sensors or other data sources and transmit it to the data preprocessing unit.

[0147] (2) Data preprocessing unit 162;

[0148] Main function: to preprocess multimodal data and input the processed data into the cross-modal fusion unit.

[0149] Hardware implementation methods include setting up multiple dedicated hardware accelerators for optimized processing of data in different modalities. For image and video data, the preprocessing unit can perform denoising, resizing, and color space conversion. For audio data, it can perform sampling rate conversion and noise reduction. For text data, it can perform operations such as word segmentation and encoding. These preprocessing operations are implemented at the hardware level, enabling rapid processing of large amounts of data and reducing latency and resource consumption in software processing.

[0150] (3) Cross-modal fusion unit 163;

[0151] Main function: Employs hardware-accelerated fusion algorithms to fuse multimodal data and obtain perception results.

[0152] Hardware implementation methods include: setting up hardware-accelerated fusion algorithm modules, such as attention mechanisms and gating units, to deeply integrate feature vectors from different modalities. A built-in optimized neural network inference core (NPU) is used to execute lightweight perceptual models such as visual recognition, speech intent recognition, and emotion detection locally on the chip. The cross-modal fusion unit 163 can fuse data from different modalities into a unified perceptual result, providing a more comprehensive and accurate understanding of the environment and supporting intelligent decision-making by the agent.

[0153] Please see Figure 2 The working principle of the above-mentioned multimodal fusion circuit 160 is as follows:

[0154] The multimodal data input interface 161 is responsible for transmitting multimodal data to the data preprocessing unit 162. The multimodal data input interface 161 supports input of various data types, including but not limited to images, video, audio, and text. The interface receives raw data from different sensors or data sources through a dedicated hardware channel and transmits it to the data preprocessing unit 162 in a high-speed, low-latency manner. Its function is similar to a data collector and transmission channel, ensuring that various types of data can smoothly enter subsequent processing stages.

[0155] The data preprocessing unit 162 preprocesses the received multimodal data. The preprocessing includes noise removal, data format adjustment, and normalization, aiming to improve data quality and consistency and prepare it for subsequent fusion processing. The data preprocessing unit 162 uses dedicated hardware circuitry to achieve efficient image denoising, audio noise reduction, and text segmentation, and transmits the preprocessed data to the cross-modal fusion unit 163.

[0156] The cross-modal fusion unit 163 employs a hardware-accelerated fusion algorithm to fuse multimodal data, ultimately obtaining a final abstract perceptual result. The cross-modal fusion unit 163 utilizes hardware-accelerated attention mechanisms and gating units to deeply integrate feature vectors from different modalities. Its built-in neural network inference core (NPU) can quickly execute lightweight perceptual models such as visual recognition, speech intent recognition, and emotion detection, thereby fusing multimodal data into a unified perceptual result. Furthermore, the cross-modal fusion unit 163 can also de-identify sensitive data during multimodal data fusion, thereby obtaining a de-identified abstract perceptual result.

[0157] The multimodal fusion circuit 160 in the above scheme can provide the chip 100 with richer environmental information by efficiently integrating multimodal data such as images, videos, audio, and text, enabling the intelligent agent to have a more comprehensive and in-depth understanding of complex scenes. On the other hand, the multimodal fusion circuit 160 accelerates data processing and fusion at the hardware level, which is conducive to improving the multimodal data fusion time from data input to perception result output, and thus improving the real-time response capability of the chip 100. Furthermore, by completing data processing locally on the chip, the multimodal fusion circuit 160 reduces the dependence on external transmission, thereby reducing the risk of data leakage and better protecting user privacy and data security.

[0158] Example 3:

[0159] Optionally, based on the above embodiment 1, the chip 100 further includes: a vector library retrieval circuit 170 connected to the execution controller 130, the vector library retrieval circuit 170 including an embedding generation unit 171 and a vector retrieval unit 172 connected in sequence, wherein:

[0160] Embedding generation unit 171 is configured to embed and generate high-dimensional vectors from input information;

[0161] The vector retrieval unit 172 is configured to retrieve associated vectors from the local vector library based on high-dimensional vectors and send the associated vectors to the execution controller 130.

[0162] The local vector library is a hardware storage module within the aforementioned chip 100, used to store high-dimensional vector data. These high-dimensional vectors can be feature vectors extracted from preprocessed data or vector representations imported from external knowledge bases. The local vector library is typically integrated within the chip, featuring low latency, high bandwidth, and security. It utilizes a dedicated storage architecture, such as on-chip SRAM or flash memory, to ensure fast access and data security.

[0163] The embedding generation unit 171 in the aforementioned vector library retrieval circuit 170 is used to convert input information (such as text, images, or other data) into a high-dimensional vector representation. The vector retrieval unit 172 is responsible for efficiently retrieving vectors similar to or related to the input vector from the local vector library. Using hardware-accelerated algorithms (such as cosine similarity calculation implemented with a SIMD architecture), the input vector is quickly compared with the vectors in the library. After finding the most similar candidate set, it is directly sent to the execution controller 130 via the bus.

[0164] Please see Figure 3When both Embodiments 1 and 3 are implemented simultaneously, the working principle of the vector library retrieval circuit 170 is as follows: The embedding generation unit 171 receives high-level task requests or contextual information from the intelligent agent and encodes this information into high-dimensional vectors. The embedding generation unit 171 can segment and encode the input text, converting it into semantic vectors, while managing vector caching and index updates to ensure data timeliness and retrieval efficiency. The vector retrieval unit 172 uses hardware acceleration to perform similarity calculations, comparing the high-dimensional vectors with vectors in the local vector library to quickly find the most similar entries. The local vector library securely stores structured information such as knowledge graphs and historical data. The retrieved related knowledge is then transmitted to the execution controller 130 for use in assisting reasoning, decision-making, or behavior generation.

[0165] It is understandable that the associated vectors retrieved by the aforementioned vector library retrieval circuit 170 can be used for knowledge enhancement. Knowledge enhancement refers to providing real-time contextual support for the execution of intelligent agent tasks by retrieving semantic information (such as knowledge base, historical records, or domain-specific data) from the local vector library. The vector library retrieval circuit 170 extracts high-dimensional vectors from the input information and finds similar entries in the local vector library, integrating these associated knowledge into the task execution process. This process can help the chip utilize rich semantic information when making decisions, thereby improving the depth and accuracy of task processing. The effects that the aforementioned chip 100 can obtain by using associated vectors for knowledge enhancement include at least the following: (1) Improved decision quality: Knowledge enhancement can provide the aforementioned chip 100 with more comprehensive contextual information, enabling it to make more informed decisions based on a richer knowledge base. (2) Enhanced semantic understanding: By combining the retrieved relevant knowledge, the aforementioned chip 100 can more accurately understand the semantics and background of the task, thereby better executing the task. (3) Support for complex task processing: When processing complex tasks, knowledge enhancement can provide necessary background knowledge and relevant information, helping the aforementioned chip 100 to solve problems more effectively. (4) Real-time performance and adaptability: Knowledge enhancement enables the chip 100 to adjust its task execution strategy based on the knowledge retrieved in real time, thereby improving its adaptability to dynamic environments.

[0166] The vector library retrieval circuit 170 in the above scheme can quickly generate high-dimensional vector representations of input information and efficiently retrieve related knowledge in the local vector library, providing rich semantic information support for task execution. On the other hand, the vector library retrieval circuit 170 enables the chip 100 to complete knowledge retrieval and enhancement locally, avoiding the delay and leakage risks of external data transmission and ensuring data privacy and security. Furthermore, the hardware acceleration architecture of the vector library retrieval circuit 170 achieves millisecond-level vector retrieval, supporting the chip to perform real-time knowledge enhancement in complex tasks, improving the accuracy and adaptability of task execution.

[0167] The aforementioned embedding generation unit 171 can obtain high-dimensional vectors through at least one or more of the following three methods:

[0168] The first method of acquisition: directly using the on-chip lightweight inference engine 140;

[0169] In this approach, the embedded generation unit 171 can directly call the on-chip lightweight inference engine 140, take the input information as its input, and after processing by the on-chip lightweight inference engine 140, directly output the corresponding high-dimensional vector.

[0170] The second method of acquisition: using a local lightweight embedded model set inside chip 100;

[0171] In this approach, a lightweight embedding model specifically designed for generating embeddings can be integrated within chip 100. This model can be a pre-trained neural network or other machine learning model. Input information is directly fed into the local lightweight embedding model, and after computation, it outputs a high-dimensional vector.

[0172] The above solution utilizes a local lightweight embedding model to generate high-dimensional vectors, which can reduce dependence on external services, reduce data transmission latency, and improve the response speed of the chip 100. On the other hand, the input information does not need to leave the chip, thereby reducing the risk of data leakage. Furthermore, the local lightweight embedding model can operate efficiently in resource-constrained environments, reducing the chip's power consumption and hardware requirements, making it suitable for more application scenarios and improving the adaptability of the chip 100.

[0173] The third method of acquisition: Obtaining the information via secure communication interface 150 using an external inference service;

[0174] In this method, the embedding generation unit 171 can send input information to the secure communication interface 150. The secure communication interface 150 performs de-identification processing on the input information and sends it to the external inference service through an encrypted channel. After receiving the de-identified input information, the external inference service uses its powerful processing capabilities to generate a high-dimensional vector and returns the result to the secure communication interface 150. After receiving the returned high-dimensional vector, the secure communication interface 150 sends it to the embedding generation unit 171.

[0175] The above solution effectively protects user privacy and sensitive data by desensitizing the input information through the secure communication interface 150, and reduces the risk of data leakage during transmission. On the other hand, by utilizing the powerful computing capabilities and advanced model architecture of the external inference service, higher quality and more accurate high-dimensional vectors can be generated, improving the accuracy and depth of embedding generation. Furthermore, it enables the chip 100 to flexibly expand its processing capabilities and call external resources on demand, which is beneficial to improving the applicability of the chip 100.

[0176] Example 4:

[0177] Optionally, based on the above embodiment 1, the chip 100 further includes: a condition judgment and branch execution circuit 180 connected to the execution controller 130, wherein:

[0178] The condition judgment and branch execution circuit 180 is configured to perform Boolean logic operations and generate branch decisions based on at least one of the following: chip internal state, perception results, return data obtained from tool call interface, external environment, and knowledge enhancement results, so as to determine the task branch to be executed and the microtask sequence corresponding to the task branch in the task graph.

[0179] The aforementioned condition judgment and branch execution circuit 180 is a key hardware module in chip 100, responsible for performing Boolean logic operations based on various input conditions to generate branch decisions, thereby determining the task branches to be executed in the task graph and the corresponding micro-task sequences. The main hardware components of the condition judgment and branch execution circuit 180 include: (1) Input condition receiving unit: receiving input data from the chip's internal state, multimodal perception results, tool call return data, external environmental sensors, and knowledge enhancement results, and using these data as judgment conditions. (2) Boolean logic operation unit: containing multiple logic gates (such as AND, OR, NOT) and comparators for performing Boolean logic operations. (3) Status register: storing the current judgment conditions and intermediate results to ensure the fast execution of logic operations. (4) Decision generation unit: generating branch decision signals based on the logic operation results to indicate the next task branch to be executed in the task graph. (5) Control signal output unit: sending the branch decision signals to the execution controller 130 to adjust the task execution flow.

[0180] When Embodiments 1, 2, 3, and 4 are implemented simultaneously, the working principle of the condition judgment and branch execution circuit 180 is as follows: The input condition receiving unit collects condition signals from multiple sources, including the chip's internal state, multimodal perception results, tool call return data, external environment sensor data, and knowledge enhancement results. The Boolean logic operation unit performs operations on the input conditions according to predefined logic rules (such as AND, OR, and NOT combinations) to generate intermediate logic results. The intermediate logic results are stored in a status register for use in subsequent operations, ensuring the continuity and speed of the judgment process. The decision generation unit generates a branch decision signal based on the final logic operation result, determining the next task branch to be executed in the task graph and the corresponding microtask sequence. The control signal output unit sends the branch decision signal to the execution controller 130, which adjusts the execution flow of the task graph according to the signal, ensuring dynamic adjustment and efficient execution of the task.

[0181] Of course, the above embodiment 4 can also be implemented simultaneously with embodiment 2 and embodiment 3. Its working principle is similar to that of the above condition judgment and branch execution circuit 180. The difference is that the conditions obtained by the condition judgment and branch execution circuit 180 are different. The specific working principle will not be repeated in this embodiment.

[0182] The following sections describe the data sources for the chip's internal state, multimodal perception results, tool call return data, external environmental sensor data, and knowledge enhancement results:

[0183] (1) Internal state of the chip: mainly derived from modules such as the execution controller 130, task graph parsing and scheduling circuit 110, and microinstruction decoding and execution circuit 120. The state data mainly includes: the execution progress of the current task, the busy / idle status of each execution core, the length of the task queue, the cache occupancy, and the resource allocation. This state information reflects the current operating status of the chip and is of great significance for the chip's dynamic scheduling and resource allocation.

[0184] (2) Multimodal perception results: mainly from the multimodal fusion circuit 160. The multimodal perception results can provide the chip 100 with a comprehensive perception of the external environment. The chip can make corresponding decisions and actions based on these results.

[0185] (3) Tool call return data: mainly from the tool call interface. The data returned by the interface mainly includes: query results, operation results, calculation results, etc. returned by external tools (such as API, database, RPA robot, etc.). This returned data provides the chip with the functional support of external tools, enabling the chip to complete more complex tasks with the help of external tools.

[0186] (4) External environment data: mainly from various sensor devices outside the chip, including environmental data collected by temperature sensors, humidity sensors, pressure sensors, light sensors, position sensors, etc. This sensor data provides the chip with real-time monitoring of the external physical environment, and the chip can make corresponding responses and adjustments based on this data.

[0187] (5) Knowledge enhancement results: These mainly come from the vector library retrieval circuit 170 and include knowledge, information, and experience related to the current task retrieved from the local vector library or external knowledge base. These knowledge enhancement results provide the chip with richer background knowledge and decision-making basis, enabling the chip to make more informed decisions.

[0188] The condition judgment and branch execution circuit 180 in the above scheme can perform Boolean logic operations in real time according to various input conditions to generate accurate branch decisions, enabling the chip 100 to dynamically adjust task branches during task execution, adapt to complex and ever-changing environments, and enhance the flexibility and adaptability of the chip 100. On the other hand, by implementing condition judgment and branch execution at the hardware level, it is beneficial to improve the decision-making speed and efficiency of the chip 100. Furthermore, by integrating the condition judgment and branch execution circuit 180 inside the chip, the dependence on external components or software is reduced, which is beneficial to improving the overall performance and reliability of the chip 100.

[0189] To facilitate understanding of the working principle of the chip 100, a specific application example of the chip 100 in a certain application scenario is provided below:

[0190] Please see Figure 4 In this application scenario, the aforementioned chip 100 mainly includes: a task graph parsing and scheduling circuit 110, a micro-instruction decoding and execution circuit 120, an execution controller 130, an on-chip lightweight inference engine 140, a secure communication interface 150, a multi-modal fusion circuit 160, a vector library retrieval circuit 170, a conditional judgment and branch execution circuit 180, an agent task input / output interface 191, and a tool invocation interface 192. The main working process of the aforementioned chip 100 can include the following stages:

[0191] (1) Task reception and initial processing;

[0192] Task Input: The agent task input / output interface 191 receives high-level task requests from users or the system. These requests can be natural language instructions, voice commands, or other forms of task requirements.

[0193] Task Conversion: Received high-level task requests are first sent to the on-chip lightweight inference engine 140. If the task complexity is low, the engine directly converts it into a task graph; if the task is complex or requires deep inference, it is sent to an external inference service via the secure communication interface 150, where the task graph is obtained after de-identification processing.

[0194] (2) Task analysis and scheduling;

[0195] The task graph parsing and scheduling circuit 110 parses the task graph, identifies task nodes and dependencies, and performs parallel scheduling to prepare for subsequent execution.

[0196] (3) Microinstruction generation and execution;

[0197] Microinstruction conversion: The microinstruction decoding and execution circuit 120 converts the parsed task node into a specific microinstruction.

[0198] Instruction execution: Microinstructions are then assigned to idle execution cores for processing to complete specific calculations or operations.

[0199] (4) Multimodal perception and knowledge enhancement;

[0200] Multimodal data processing: The multimodal fusion circuit 160 processes multimodal data from sensors in real time and fuses them into a unified perception result for reference in task execution.

[0201] Knowledge retrieval: The vector library retrieval circuit 170 retrieves local or external knowledge bases according to needs to obtain relevant knowledge to enhance the decision-making ability for task execution.

[0202] (5) Conditional judgment and branch execution;

[0203] The condition judgment and branch execution circuit 180 dynamically generates branch decisions based on chip status, sensing results, etc., and adjusts the task execution path to ensure that the task adapts to real-time changes.

[0204] (6) Tool calls and external interactions;

[0205] Tool call interface 192 calls external tools or services when needed to obtain additional support or data, and continues task execution after the result is returned.

[0206] (7) Execution control and result output;

[0207] Process control: The execution controller 130 runs through the entire process, coordinating the operation of various circuits and modules, ensuring that tasks are executed as planned, and handling exceptions.

[0208] Output results: After the task is completed, the final result is returned to the user or system via the agent task input / output interface 191.

[0209] Through this highly integrated and collaborative architecture, the aforementioned chip 100 can efficiently, flexibly, and securely perform complex intelligent agent tasks, adapting to changing environments and needs.

[0210] Based on the same inventive concept, this application also provides an LLM-based intelligent agent system, including an external inference service and the aforementioned chip 100, wherein the chip 100 communicates with the external inference service through a secure communication interface 150.

[0211] Based on the same inventive concept, this application also provides an intelligent agent device, including the chip 100 described above.

[0212] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0213] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0214] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0215] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A chip, characterized in that, The chip includes: a task graph parsing and scheduling circuit, a microinstruction decoding and execution circuit, and an execution controller. The task graph parsing and scheduling circuit and the microinstruction decoding and execution circuit are respectively connected to the execution controller, wherein: The task graph parsing and scheduling circuit is configured to parse the received task graph, obtain the micro-tasks to be executed, and send the micro-tasks to be executed to the micro-instruction decoding and execution circuit. The microinstruction decoding and execution circuit is configured to convert the received microtask into microinstructions and allocate the microinstructions to an idle execution core so that the execution core executes the microinstructions; The execution controller is configured to coordinate the operation of the circuits in the chip to complete the microtask; The chip further includes: a secure communication interface connected to the execution controller, wherein: The secure communication interface is configured to de-identify high-level task requests through an encrypted channel, and send the de-identified high-level task requests to an external inference service to obtain the task graph generated by the external inference service based on the de-identified high-level task requests.

2. The chip according to claim 1, characterized in that, The task graph parsing and scheduling circuit includes: a multi-parallel hardware scheduler, wherein: The hardware scheduler, which operates in a multi-path parallel manner, is configured to perform parallel parsing and scheduling of the task graph.

3. The chip according to claim 1, characterized in that, The microinstruction decoding and execution circuit includes: a microinstruction decoder and multiple execution cores, wherein: The micro-instruction decoder is configured to convert the received micro-task into the micro-instruction and send the micro-instruction to the corresponding execution core; The execution core is configured to receive and execute the microinstructions.

4. The chip according to any one of claims 1 to 3, characterized in that, The chip further includes: a multimodal fusion circuit connected to the execution controller, the multimodal fusion circuit comprising a multimodal data input interface, a data preprocessing unit, and a cross-modal fusion unit connected in sequence, wherein: The multimodal data input interface is configured to transmit multimodal data to the data preprocessing unit; The data preprocessing unit is configured to preprocess the multimodal data and input the preprocessed multimodal data into the cross-modal fusion unit. The cross-modal fusion unit is configured to use a hardware-accelerated fusion algorithm to fuse the multimodal data and obtain the perception results.

5. The chip according to claim 4, characterized in that, The chip further includes: a condition judgment and branch execution circuit connected to the execution controller, wherein: The condition judgment and branch execution circuit is configured to perform Boolean logic operations and generate branch decisions based on at least one of the following: chip internal state, the perception result, the return data obtained from the tool call interface, the external environment, and the knowledge enhancement result, so as to determine the task branch to be executed and the microtask sequence corresponding to the task branch in the task graph.

6. The chip according to any one of claims 1 to 3, characterized in that, The chip also includes: an on-chip lightweight inference engine connected to the execution controller, wherein: The on-chip lightweight inference engine is configured to convert received high-level task requests into the task graph.

7. The chip according to any one of claims 1 to 3, characterized in that, The chip further includes: a vector library retrieval circuit connected to the execution controller, the vector library retrieval circuit comprising an embedding generation unit and a vector retrieval unit connected in sequence, wherein: The embedding generation unit is configured to embed the input information to obtain a high-dimensional vector; The vector retrieval unit is configured to retrieve associated vectors from the local vector library based on the high-dimensional vector, and send the associated vectors to the execution controller.

8. The chip according to claim 7, characterized in that, The embedding generation unit is configured to: The high-dimensional vector is obtained by embedding the input information using a local lightweight embedding model.

9. The chip according to claim 7, characterized in that, The embedding generation unit is configured to: The input information is de-identified using a secure communication interface and then sent to an external inference service to obtain the high-dimensional vector output by the external inference service.

10. An LLM-based intelligent agent system, characterized in that, It includes an external inference service and a chip as described in any one of claims 1 to 9, wherein the chip communicates with the external inference service via a secure communication interface.

11. An intelligent agent device, characterized in that, Includes the chip as described in any one of claims 1 to 9.

Images