Intelligent agent system, data processing method thereof and related equipment

Abstract

The invention provides an agent system, a data processing method thereof and related equipment, and relates to the technical field of artificial intelligence. The intelligent agent system comprises a first system, a second system and a third system, the first system converts the sensed environment state data into structured environment sensing information which can be identified and / or processed in the system, and executes an action command issued by the second system; the second system performs task reasoning decomposition on the target task plan issued by the third system based on an agent strategy model of a large language model, generates an action command executable by an agent, and issues the action command to the first system; and the third system generates a target task plan and mixed reward information used for updating the agent strategy model, and updates the agent autonomous model according to historical interaction information of the agent and the environment and related information in the execution process of the target task plan. According to the invention, the intelligent agent system with identity continuity and long-term state maintenance can be provided.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, and in particular, to an agent system and a data processing method thereof and related equipment. BACKGROUND

[0002] With the development of Large Language Model (LLM) technology, artificial intelligence is evolving from "completing discrete tasks" to "long-term autonomous operation". The demand for agent systems in the industry is not only to complete a single task, but also to expect the agent system to have biological characteristics such as long life cycle, autonomous learning, self-evolution, consistency of emotion and motivation.

[0003] Currently, the agent system provided in the related art is task-centered rather than agent-centered. Once the conversation or task is completed, the experience and knowledge learned and self-cognition cannot be persisted and inherited across cycles, resulting in a "newborn" each time the agent is started, and a stable narrative identity cannot be formed.

[0004] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0005] The present disclosure provides an agent system and a data processing method thereof and related equipment, which at least partially overcome the technical problems of lack of identity continuity and long-term state retention of the agent system provided in the related art.

[0006] Other characteristics and advantages of the present disclosure will become apparent from the following detailed description, or will be learned by practice of the present disclosure.

[0007] According to one aspect of the present disclosure, an agent system is provided, comprising: a first system interacting with an external environment, a second system deploying an agent policy model, and a third system deploying an agent autonomous model; wherein the first system is configured to convert perceived environment state data into structured environment perception information recognizable and / or processable within the agent system, transmit to the third system, and execute action commands issued by the second system; the second system is configured to perform task reasoning decomposition on the target task plan issued by the third system based on the agent policy model of the large language model, generate action commands executable by the agent, and issue to the first system; the third system is configured to generate a target task plan and hybrid reward information for updating the agent policy model, issue to the second system, and update the agent autonomous model according to historical interaction information between the agent and the environment, structured environment perception information, reasoning context information, agent action information, and task execution result during execution of the target task plan, wherein the agent autonomous model is used for the agent system to execute autonomous learning task plan, and the hybrid reward information is a fusion result of task reward information and intrinsic reward information within the system.

[0008] In some embodiments, the first system comprises: a multi-modal perception encoder and an action executor; wherein the multi-modal perception encoder is configured to convert perceived environment state data of multiple modalities into structured environment perception information recognizable and / or processable within the agent system; and the action executor is configured to execute action commands issued by the second system.

[0009] In some embodiments, the second system comprises: a large language model planner and a policy update module; wherein the large language model planner is configured to perform task reasoning decomposition on the target task plan issued by the third system based on the agent policy model of the large language model, and generate action commands executable by the agent; and the policy update module is configured to update the agent policy model according to the hybrid reward information generated by the third system.

[0010] In some embodiments, the third system comprises: a meta-cognition monitor, a memory module, a user model, an agent autonomous model, and a hybrid reward module; wherein the meta-cognition monitor is configured to generate a target task plan and issue it to the second system; the memory module is configured to store structured environment perception information, reasoning context information, agent action information, and task execution results in the process of the target task plan in a vector database and / or a graph database; the user model is configured to establish dynamic and updated user state information for each user; the agent autonomous model is configured to compare the capability information of the agent system with the task demand in response to an event trigger of generating the target task plan or completing the target task plan, and generate an autonomous learning task plan for the agent system according to the comparison result; and the hybrid reward module is configured to generate hybrid reward information for updating the agent policy model according to the task reward information provided by the second system and the internal reward information generated by the agent system.

[0011] According to another aspect of the present disclosure, a data processing method of an agent system is also provided, comprising: obtaining a target task plan to be executed; performing task reasoning decomposition on the target task plan based on an agent policy model of a large language model to generate an action command executable by the agent; updating the agent policy model according to hybrid reward information in response to the agent executing each action command, the hybrid reward information being a fusion result of task reward information and system internal reward information; converting perceived environment state data into structured environment perception information recognizable and / or processable by the agent system; and updating an agent autonomous model based on historical interaction information of the agent and the environment and structured environment perception information, reasoning context information, agent action information, and task execution results in the process of the target task plan, the agent autonomous model being used for the agent system to execute an autonomous learning task plan.

[0012] In some embodiments, the method further comprises: structurally storing the structured environment perception information, the reasoning context information, the agent action information, and the task execution results in the process of the target task plan.

[0013] In some embodiments, after the agent policy model based on the large language model performs task reasoning decomposition on the target task plan to generate an action command executable by the agent, the method further comprises: verifying each action command output by the agent policy model based on a pre-trained process supervision model; and determining whether to execute the corresponding action command according to the verification result.

[0014] According to another aspect of the present disclosure, an electronic device is also provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the data processing method of the agent system according to any one of the above aspects via execution of the executable instructions.

[0015] According to another aspect of the present disclosure, a computer-readable storage medium having stored thereon a computer program is also provided, the computer program being executed by a processor to implement the data processing method of the agent system according to any one of the above aspects.

[0016] According to another aspect of the present disclosure, a computer program product is also provided, comprising: a computer program or instructions, the computer program or instructions being executed by a processor to implement the data processing method of the agent system according to any one of the above aspects.

[0017] The agent system and the data processing method thereof and the related device provided in the embodiments of the present disclosure, through the first system interacting with the external environment, convert the perceived environment state data into structured environment perception information recognizable and / or processable inside the agent system, through the second system deploying the agent policy model, based on the agent policy model of the large language model, perform task reasoning decomposition on the target task plan issued by the third system, generate the action command executable by the agent, and issue it to the first system, so that the first system executes the action command issued by the second system; through the third system deploying the agent autonomous model, generate the target task plan and the hybrid reward information for updating the agent policy model, and issue it to the second system, so that the second system executes the corresponding target task plan, and in the execution process, updates the agent policy model according to the hybrid reward information, and the third system can also update the agent autonomous model according to the historical interaction information between the agent and the environment, and the structured environment perception information, reasoning context information, agent action information, and task execution result in the execution process of the target task plan, so that the agent system executes the autonomous learning task plan.

[0018] Through the embodiments of the present disclosure, an agent system based on a three-layer meta-cognition architecture can be provided, which can realize identity continuity and long-term state maintenance, and can also autonomously think and generate short-term survival goals without external task commands, and make action and learning plans for itself.

[0019] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, further serve to explain the principles of the disclosure. It is to be clearly understood that the drawings are merely representative of some embodiments of the disclosure and that additional drawings can be derived from these drawings by one of ordinary skill in the art without paying creative effort.

[0021] Figure 1 Fig. 1 shows a schematic diagram of an agent system according to an embodiment of the present disclosure; Figure 2 Fig. 2 shows a schematic diagram of an alternative agent system according to an embodiment of the present disclosure; Figure 3 Fig. 3 shows a schematic diagram of an alternative agent system according to an embodiment of the present disclosure; Figure 4 Fig. 4 shows a schematic diagram of an alternative agent system according to an embodiment of the present disclosure; Figure 5 Fig. 5 shows a flow chart of a data processing method of an agent system according to an embodiment of the present disclosure; Figure 6 Fig. 6 shows a flow chart of a data processing method of an agent system according to an embodiment of the present disclosure; Figure 7 Fig. 7 shows a flow chart of a method for verifying a task command according to an embodiment of the present disclosure; Figure 8 Fig. 8 shows a framework diagram of an artificial life-like agent technology according to an embodiment of the present disclosure; Figure 9 Fig. 9 shows a structural diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] Example implementations will now be described with reference to the drawings; however, example implementations can be implemented in many different forms and should not be construed as limited to the examples set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the example implementations to those skilled in the art. The described features, structures, or characteristics can be combined in one or more implementations.

[0023] In addition, the drawings are merely schematic and are not drawn to scale. Identical or similar elements are denoted by the same reference numerals throughout the various figures. It will be readily understood that the components as given by the reference numerals need not be implemented in this way. Rather, the individual components or the respective functional means can be implemented by software or hardware or a hybrid of both.

[0024] For ease of understanding, before introducing the embodiments of the present disclosure, first, several terms involved in the embodiments of the present disclosure are explained as follows: First system (System1 / System One): located at the lowest layer of the agent system, that is, the perception and rapid reaction layer, used to process the perceived multi-modal data and execute the bottom layer action instructions.

[0025] Second system (System2 / System Two): located at the middle layer of the agent system, that is, the slow thinking and deep reasoning layer, used for multi-step planning, problem solving and complex decision making.

[0026] Third system (System3 / System Three): located at the top layer of the agent system, that is, the meta-cognition layer, used to supervise, manage and optimize the operation of the first system and the second system, maintain the identity of the agent, generate goals and drive self-improvement.

[0027] Meta-cognition: refers to the cognition of cognition, that is, the high-order cognitive ability of monitoring, evaluating and adjusting the thinking process of oneself. It is the theoretical basis of the third system.

[0028] Theory of Mind: refers to the ability to infer the psychological state (such as belief, intention, knowledge, emotion) of others (such as users). In the embodiments of the present disclosure, this function is realized by the user model.

[0029] Intrinsic motivation: refers to the driving force generated internally by the agent (such as curiosity, sense of control), rather than driven by external rewards. It is the key driving force for the agent to actively explore and self-evolve, calculated by the mixed reward module.

[0030] Episodic memory: autobiographical memory of events and their context information occurring at a specific time and place. In the embodiments of the present disclosure, this function is realized by the memory module, providing identity continuity and experience basis for the agent.

[0031] The specific embodiments of the embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0032] At present, the agent system provided in the related art mainly includes the following categories: A. Prompt-based instant dialogue system.

[0033] Typical products / technologies: ChatGPT, Claude, etc. General chat robots.

[0034] Technical mechanism: Relying on the strong contextual understanding ability of large language models, it reasons and responds within a single session or limited short window context. Its state is transient and lost after the session ends, without the ability to maintain state and memory across sessions.

[0035] B. Tool orchestration LLM agent framework.

[0036] Representative projects: AutoGPT, LangChain Agent, BabyAGI.

[0037] Technical mechanism: A "plan-execute-reflect" cycle mechanism is built. Through LLM, task decomposition and plan generation are performed, external tools (such as search engines, APIs) are called to execute subtasks, and the results are briefly evaluated. Its life cycle is usually bounded by the completion of a single complex task input by the user, and the agent state is reset after the task is completed.

[0038] Limitations: Lack of motivation for long-term continuous operation and identity consistency, unable to autonomously generate goals and self-improve outside of tasks.

[0039] C. Continual / Lifelong Learning (CL) method.

[0040] Technical mechanism: Aimed at enabling the model to continuously learn new knowledge in sequential tasks while mitigating catastrophic forgetting of old knowledge. Main methods include: Architecture method (Parameter Isolation / Expansion): Assign new model parameters or modules for new tasks. Regularization method (Regularization-based): Restrict weight updates to protect parameters important to old tasks. Memory replay (Memory Replay): Maintain a sample buffer of old tasks and replay them when learning new tasks.

[0041] Application in the field of LLM: Continual pre-training, instruction tuning, and alignment strategies.

[0042] Limitations: This type of method usually passively accepts externally defined task sequences and data distributions, lacking the ability to autonomously discover goals and self-direct learning.

[0043] As can be seen, the agent systems provided in the related art have the following inherent shortcomings, making it difficult to achieve a truly "artificial life" agent: 1) Lack of Identity Continuity and Long-term State Persistence. The reason is that the existing architecture (Classes A and B) is task-centered rather than agent-centered. After the session or task is over, its memory, learned experience, and self-awareness cannot be persisted and inherited across cycles, resulting in a "newborn" every time it starts, and it cannot form a stable narrative identity.

[0044] 2) Passive vs. Active Learning. The reason is that existing continuous learning techniques (Class C) and learning mechanisms rely on external carefully planned task scheduling and dataset input. The agent itself cannot autonomously identify knowledge gaps, generate learning goals, or develop its own learning curriculum. Essentially, it is "taught" rather than "self-taught," limiting its adaptability and evolution ability in an open environment.

[0045] 3) Lack of Meta-Cognition, unable to audit and improve its reasoning process. The reason is that existing architectures (especially Classes A and B) lack a high-level supervision layer to monitor, evaluate, and correct the reasoning process of System 1 and System 2 in real time. This results in the inability to discover logical fallacies in its own thinking process, balance short-term rewards and long-term goals, and ensure consistency between behavior and internal values (Creed), posing potential security risks and strategic short-sightedness.

[0046] 4) High running cost and coarse-grained scheduling mechanism. The reason is that to achieve complex reasoning, existing agent frameworks frequently call large LLMs for every step of the thought chain derivation, resulting in extremely high computational cost. The lack of a hierarchical reasoning scheduling mechanism (such as pre-screening or state judgment with lightweight models, triggering deep reasoning only when necessary) results in uncontrolled cloud deployment costs and difficulty in supporting large-scale long-term deployment.

[0047] 5) Single motivation system, lack of "life sense". The reason is that the behavior of existing systems is completely driven by external task rewards (Extrinsic Reward), and lacks internal motivation (Intrinsic Motivation) mechanisms such as curiosity, sense of control, and social relevance. This makes the agent's behavior utilitarian and short-sighted, lacking the internal exploration and growth drive of biological organisms, and difficult to embody "life sense."

[0048] It can be seen that the above defects jointly cause the current technology to be difficult to meet the core needs of "artificial life-like" agents with continuous autonomy, evolution, and emotional consistency.

[0049] To solve the above problems, an artificial life-like agent-oriented three-layer meta-cognitive architecture (including system one, system two, and system three) with continuous self-learning ability and its cooperative working mechanism are provided in the embodiments of the present disclosure. The architecture solves the fundamental problem of the lack of identity continuity, autonomy, and long-term evolution ability of existing agents by computing psychological concepts. Specifically, it includes the following aspects: 1) The overall architecture and execution monitoring mechanism of system three (meta-cognitive layer): Core: Above the traditional perception (system one) and reasoning (system two) layers, a meta-cognitive monitoring layer (system three) with the highest decision-making authority is added, as well as its internal integration of three core mechanisms: thought search, process supervision, and reflection.

[0050] Innovation: The architecture changes the agent from an "execution tool" to a "cognitive subject", which can supervise, evaluate, and actively optimize its underlying reasoning process. This is the technical basis for realizing the "self-awareness" and "self-improvement" capabilities of life-like agents.

[0051] 2) Modular engineering implementation based on four psychological pillars: Core: Four abstract psychological concepts: meta-cognition, mental theory, intrinsic motivation, and episodic memory are specifically transformed into four computable and interactive software modules: self-model, user-model, hybrid reward module, and memory module, and integrated under system three.

[0052] Innovation: Through the organic cooperation between modules, the agent's identity continuity (Self+Memory), social interaction (User-Model), and self-driven force (Hybrid Reward) are first integrated in engineering, forming the basis of the "personality" of life-like agents.

[0053] 3) Self-driven learning mechanism based on hybrid rewards: Core: The hybrid reward module generates a total reward signal that combines external task rewards and intrinsic motivation (curiosity, sense of control, etc.), and introduces a balance parameter dynamically adjusted by the meta-cognitive monitor to control the agent's tendency between "exploration" and "exploitation" in real time.

[0054] Innovation: This mechanism is the key to the agent's "subjective initiative" performance, making its behavior no longer short-sighted and utilitarian, but able to actively explore for long-term capability development, which is the essential feature that distinguishes it from all task-driven agents.

[0055] 4) Reinforcement learning driven strategy optimization mechanism guided by metacognition layer: Core: The strategy updating mechanism of system two adopts reinforcement learning (RL) algorithm as its core implementation. Its unique feature is that the reward signal driving learning does not come from the external environment, but is completely generated by the hybrid reward module of system three. This signal combines external task rewards and internal motivation. System two continuously optimizes its decision-making strategy by maximizing the total reward provided by system three through RL methods such as policy gradient.

[0056] Innovation: This mechanism transforms reinforcement learning from a tool for optimizing external task performance into a tool for realizing the agent's internal value-driven self-evolution. System three plays the role of an "internal environment" that shapes the behavior of system two through reward signals, making it ultimately consistent with the agent's long-term goals, beliefs, and motivations, and is the core engine for achieving autonomous evolution.

[0057] Figure 1 An agent system schematic diagram in the embodiment of the present disclosure is shown, as Figure 1 shown, the agent system includes a first system 10 (i.e. system one) interacting with the external environment, a second system 20 (i.e. system two) deploying an agent strategy model, and a third system 30 (i.e. system three) deploying an agent autonomous model.

[0058] Among them, the first system 10 is used to convert the perceived environment state data into structured environment perception information recognizable and / or processable inside the agent system, and transmit it to the third system 30, and execute the action command issued by the second system 20; the second system 20 is used to perform task reasoning decomposition on the target task plan issued by the third system 30 based on the agent strategy model of the large language model, generate the action command executable by the agent, and issue it to the first system 10; the third system 30 is used to generate the target task plan and the hybrid reward information for updating the agent strategy model, and issue it to the second system 20, update the agent autonomous model according to the historical interaction information between the agent and the environment, and the structured environment perception information, reasoning context information, agent action information, and task execution result in the target task plan execution process, the agent autonomous model is used for the agent system to execute the autonomous learning task plan, and the hybrid reward information is the fusion result of the task reward information and the internal reward information of the system.

[0059] Exemplarily, in some embodiments, as Figure 2As shown, the first system 10 includes a multi-modal perception encoder 101 and an action executor 102; the multi-modal perception encoder 101 is configured to convert the perceived multi-modal environment state data into structured environment perception information recognizable and / or processable by the agent system internally; and the action executor 102 is configured to execute the action command issued by the second system 20.

[0060] Exemplarily, in some embodiments, as shown in Figure 3 As shown, the second system 20 includes a large language model planner 201 and a policy updating module 202; the large language model planner 201 is configured to perform task reasoning decomposition on the target task plan issued by the third system based on the agent policy model of the large language model, to generate an action command executable by the agent; and the policy updating module 202 is configured to update the agent policy model according to the hybrid reward information generated by the third system.

[0061] It should be noted that the policy updating of the second system 20 can be but is not limited to reinforcement learning (RL) algorithm. Alternative solutions include but are not limited to: ① Supervised learning is adopted: when the reflection process of the third system produces high-quality decision examples, these examples can be used to fine-tune the LLM planner of the second system 20, so that it directly learns the excellent decision mode. ② Evolutionary algorithm is adopted: different decision strategies are regarded as individuals, and are screened and crossed and mutated through the reward provided by the third system, to realize the iterative optimization of the strategy.

[0062] In the embodiments of the present disclosure, the lightweight deployment of the second system 20: to cope with the edge environment (such as mobile phones, embedded devices) with limited computing resources, the functions of the second system 20 can be realized by a series of carefully designed small models or rule engines, and the distillation technology is used to obtain part of the reasoning ability of the large LLM.

[0063] Exemplarily, in some embodiments, as shown in Figure 4As shown, the third system 30 includes a meta-cognition monitor 301, a memory module 302, a user model 303, an autonomous agent model 304, and a hybrid reward module 305. The meta-cognition monitor 301 is configured to generate a target task plan and issue the target task plan to the second system. The memory module 302 is configured to store structured environment perception information, reasoning context information, agent action information, and task execution results in a process of the target task plan based on a vector database and / or a graph database. The user model 303 is configured to establish dynamic user state information for each user. The autonomous agent model 304 is configured to compare capability information of the agent system with task requirements in response to an event trigger of generating the target task plan or completing the target task plan, and generate an autonomous learning task plan of the agent system according to a comparison result. The hybrid reward module 305 is configured to generate hybrid reward information for updating the agent policy model based on task reward information provided by the second system and internal reward information generated by the agent system.

[0064] It should be noted that the underlying storage and retrieval technology of the memory module can not be limited to the combination of the vector database and / or the graph database. Alternative solutions include but are not limited to: ① using a relational database or a non-relational database (such as MongoDB) in combination with a semantic retrieval middleware to achieve similar functions. ② Using a knowledge graph as the core storage medium, which focuses more on the structured relationship between entities. ③ Using a hierarchical memory system to store very short-term, short-term, and long-term memories with different technologies (such as cache, database) to improve access efficiency.

[0065] In the embodiments of the present disclosure, the meta-cognition monitor and the functional modules of the third system 30 can not be limited to centralized deployment. Distributed deployment can be considered: the user model, the memory module, and the like are deployed as independent microservices, and the meta-cognition monitor cooperates through service calls to improve the scalability and robustness of the system.

[0066] The above-mentioned extensions and alternatives prove that the three-layer meta-cognition architecture proposed in the embodiments of the present disclosure has high flexibility and universality. The core innovation point lies in the top-level design idea rather than being bound to a specific underlying implementation technology.

[0067] Based on the same inventive concept, the embodiments of the present disclosure also provide a data processing method of an agent system, which can be executed by any electronic device with computing processing capability.

[0068] In some embodiments, the data processing method of the agent system provided in the embodiments of the present disclosure can be executed by a terminal device; in other embodiments, the data processing method of the agent system provided in the embodiments of the present disclosure can be executed by a server; in other embodiments, the data processing method of the agent system provided in the embodiments of the present disclosure can be realized by the terminal device and the server through interaction.

[0069] It should be noted that the medium providing the communication link between the terminal device and the server can be a wired network or a wireless network. Optionally, the wireless network or the wired network uses standard communication technology and / or protocol. The network is usually the Internet, but can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of virtual private networks. In some embodiments, technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML) and the like are used to represent the data exchanged through the network. In addition, all or some links can be encrypted using conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPSec) and the like. In other embodiments, custom and / or dedicated data communication technologies can be used instead of or in addition to the above data communication technologies.

[0070] Illustratively, the terminal device herein can be various electronic devices, including but not limited to a smartphone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a wearable device, an augmented reality device, a virtual reality device, and the like. Optionally, the client of the application installed in different terminal devices is the same, or the client of the same type of application based on different operating systems. Based on the difference of the terminal platform, the specific form of the client of the application can also be different, for example, the client of the application can be a mobile phone client, a PC client, and the like.

[0071] Exemplarily, the server herein can be a server providing various services, for example, a background management server providing support for the operation of the device by the user using the terminal equipment. The background management server can analyze and process the received request and other data, and feed back the processing result to the terminal equipment. Alternatively, the server can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platform.

[0072] Figure 5 A flow chart of a data processing method of an agent system in an embodiment of the present disclosure is shown in FIG. 5. Figure 5 As shown in FIG. 5, the data processing method can include the following steps: S502, obtaining a target task plan to be executed.

[0073] It should be noted that in the embodiments of the present disclosure, the target task plan is a task plan that needs to be completed to achieve a certain task requirement. The target task plan is generated based on structured environment perception information, which enables the agent to actively generate a learning plan without relying on the outside world. In some embodiments, the agent system can also autonomously generate the target task plan.

[0074] S504, based on the agent policy model of the large language model, performing task reasoning decomposition on the target task plan to generate an action command executable by the agent.

[0075] In the embodiments of the present disclosure, the agent policy model is a policy model for controlling the agent to execute the action command. Considering that the target task plan often needs multiple steps to be completed, in the embodiments of the present disclosure, the agent policy model based on the large language model performs task reasoning decomposition on the target task plan, and outputs one or more action commands to be executed.

[0076] S506, in response to the agent executing each action command, updating the agent policy model according to the hybrid reward information, which is the fusion result of the task reward information and the system intrinsic reward information.

[0077] In specific implementation, an external task reward can be generated after each task command is executed, and the external task reward is fused with the internal reward of the agent system to obtain a hybrid reward information, and then the hybrid reward information is used to update the agent policy model. Since the external task reward is an objective feedback from the environment, and the internal reward is a subjective motivation (such as curiosity, sense of control and consistency) generated by the agent system itself, the agent policy model is updated according to the hybrid reward information obtained by fusing the internal reward and the external reward, which can guide the agent to complete the task efficiently while maintaining curiosity, continuous learning and consistent behavior, similar to a real life form.

[0078] S508, convert the perceived environment state data into structured environment perception information that can be recognized and / or processed inside the agent system.

[0079] It should be noted that the above environment state data can be data that can directly reflect the environment state collected by various sensors, data collection or manual collection. The data here can be any one or more modalities of data such as text, image, audio, etc. In different task scenarios, different data may need to be collected, and in some embodiments, it can include but is not limited to one or more of the following: physical environment state data (such as environment temperature, environment humidity, etc.), spatial environment state data (such as obstacle distance, coordinate data or point cloud data of various objects in the environment, etc.), dynamic environment state data (such as the speed of movable objects in the environment, etc.), and scene data (such as the on-off state of various sensors deployed in the environment).

[0080] In the embodiments of the present disclosure, the structured environment perception information refers to information formed after the "environment state data" is cleaned, classified, associated and integrated, which has a fixed format, logical relationship and clear semantics. It can directly reflect the overall condition or key features of the environment, and can be quickly understood by algorithms, systems or humans and used for decision-making. In an embodiment, it can be but is not limited to structured data in JSON format. JSON (JavaScript Object Notation) is a lightweight data exchange format based on text format, aiming to express structured data simply and clearly, and is easy for humans to read and write, and also easy for machines to parse and generate.

[0081] S510, update the agent autonomous model according to the historical interaction information of the agent and the environment, and the structured environment perception information, the reasoning context information, the agent action information and the task execution result in the target task plan execution process. The agent autonomous model is used for the agent system to execute the autonomous learning task plan.

[0082] In the embodiments of the present disclosure, the autonomous model of the agent refers to a model used for the agent system to perform an autonomous learning task plan, which can be used to update the capability information (such as a capability list) or state information of the agent system. After each agent completes the target task plan, the autonomous model of the agent is updated according to the historical interaction information of the agent and the environment, and the structured environment perception information, reasoning context information, agent action information, and task execution result in the execution process of the target task plan, so as to realize self-learning of the agent.

[0083] It should be noted that each of the steps S502-S510 can be executed on the same device or on different devices. In an optional embodiment, the steps can be implemented by the corresponding modules of the agent system as shown in the figure. Figures 1 to 4

[0084] In some embodiments, as shown in the figure, the data processing method of the agent system provided in the embodiments of the present disclosure can further include the following steps: Figure 6 S512, the structured environment perception information, reasoning context information, agent action information, and task execution result in the execution process of the target task plan are stored in a structured manner.

[0085] Optionally, the structured environment perception information, reasoning context information, agent action information, and task execution result in the execution process of the target task plan can be stored in a structured manner through a graph database and / or a vector database.

[0086] In some embodiments, after the agent policy model based on the large language model performs task reasoning decomposition on the target task plan and generates executable action commands of the agent, the data processing method of the agent system provided in the embodiments of the present disclosure can further verify each task command output by the agent policy model through the method as shown in the figure: Figure 7 S702, based on the pre-trained process supervision model, each action command output by the agent policy model is verified; S704, whether to execute the corresponding action command is determined according to the verification result.

[0087] It should be noted that the acquisition, storage, use, processing, etc. of data in the technical solutions of the present disclosure comply with the relevant provisions of relevant laws and regulations. The personal identity data, operation data, behavior data, etc. of various types of data of individuals, customers, and crowds, etc. acquired in the embodiments of the present disclosure have obtained the consent of the user.

[0088] Figure 8 An agent technical framework diagram of a kind of artificial life in the embodiments of the present disclosure is shown, as shown in the figure Figure 8 ​​​The core of the framework is to introduce a high-level meta-cognition layer called "System 3" that mimics the core cognitive functions of self-reflection, long-term adaptation, and identity maintenance, and continuously supervises, audits, and optimizes the underlying traditional perception (System 1) and reasoning (System 2) modules.

[0089] Figure 8 In the system architecture shown, the following processes can be executed in a loop: ① Input the content of memory retrieval and the reasoning context information of the second system (System 2) into the third system (System 3); ② The third system (System 3) generates a target task plan and a mixed reward, updates the self-model (autonomous model); ③ The second system (System 2) performs reasoning decomposition on the target task plan based on the agent strategy model of the large language model, and obtains executable action commands; the first system (System 1) executes the corresponding actions; ④ The interaction information is stored in the memory module; ⑤ The second system (System 2) updates the agent strategy model; ⑥ Enter the next cycle.

[0090] For the third system (System 3), the process it executes is: ① Receive reasoning context input from the second system (System 2), combine memory retrieval, and update the self-model (autonomous model); ② Perform thought search and generate target task plan; ③ Supervise the action commands executed by the agent during the execution of the target task plan; ④ Generate short-term goals and reward model.

[0091] For the process of the second system (System 2) + first system (System 1) interacting with the environment: ① The first system (System 1) converts the perceived environment state into structured environment perception information that can be recognized and processed within the system; ② The second system (System 2) performs reasoning decomposition on the target task plan based on the agent strategy model of the large language model, and obtains executable action commands; ③ The first system (System 1) executes the corresponding actions according to the action commands; ④ Interact with the environment; ⑤ Store the interaction information in the memory module.

[0092] The agent technology framework of artificial life-like entity provided in the embodiments of the present disclosure breaks through the limitations of traditional agent systems by engineering four psychological pillars, namely meta-cognition, theory of mind, intrinsic motivation, and episodic memory, into computable technical modules, and exhibits life-like characteristics, including but not limited to the following points: ① Autonomous goal generation: autonomous thinking and generating short-term survival goals without external commands, and making action and learning plans for oneself.

[0093] ② Maintaining identity continuity: maintaining stable behavior patterns, memories, and values across sessions and task cycles, and forming a unified "narrative self".

[0094] ③ Realizing safe and transparent self-improvement: auditing the reasoning process in real time, balancing external task rewards and intrinsic motivation, and ensuring that behavior aligns with established goals and safety guidelines.

[0095] ④ Efficient and low-cost operation: reducing unnecessary calls to large models through hierarchical scheduling and memory reuse, making long-term continuous operation feasible in terms of cost.

[0096] Figure 8 The framework shown is a general technical encapsulation layer that can be integrated on top of any large language model (LLM)-based system 1 / system 2 agent stack, enabling it to evolve into an agent with artificial life-like characteristics.

[0097] 1) For Figure 8 The agent technology framework of artificial life-like entity shown in the figure is described as follows: 1.1) Perception upstream: the state of the external environment generates an observation. The multimodal perception encoder of system 1 converts the original sensor observation into a typed and timestamped internal event, and immediately publishes it to an internal message bus. Each original action performed by system 1 generates an external reward, and the signal is directly streamed to the hybrid reward module of system 3.

[0098] 1.2) Control downstream: the executive monitor of system 3 receives all events reported on the bus. It fuses signals from four functional pillars (user model, memory module, self model, and hybrid reward), executes meta-strategy, and outputs two key decisions: ① The next short-term goal: derived from the highest-value leaf node found in its thought search tree.

[0099] ②Intrinsic reward scalar and balance parameters: generated by its hybrid reward module, integrating intrinsic drivers such as curiosity, sense of control and consistency.

[0100] These two outputs are pushed downstream: the goal serves as a seed to initiate the next System 2 inference cycle; the intrinsic reward is combined with the external feedback to form the total reward, which is used to update the System 2 strategy.

[0101] 1.3) Execution & Learning Loop: The System 2 LLM planner receives the current goal, local draft memory, and encoded observation stream. It queries the LLM using thought chain cues and converts the text response into a high-level command through a parser. This command is sent to the System 1 action executor to be converted into the original action for execution. The extrinsic reward generated after execution and the intrinsic reward generated by System 3 together form the total reward, which is used to update the parameters of System 2 through policy gradients, thereby achieving learning.

[0102] 1.4) Memory & Reflection Loop: The reasoning trajectory successfully executed throughout the entire interaction cycle (structured as follows)<goal, context, chain-of-thought, outcome> The data is submitted to the memory module of System 3 for storage. Upon task termination, the metacognitive monitor initiates a reflection process to perform post-hoc analysis of the survival path, compare predictions with results, extract reusable heuristics, and update the self-model's capability list and state. These persisted memories and experiences provide historical context and knowledge base for subsequent decisions, ensuring the continuity of the agent's identity and the cumulative growth of its capabilities.

[0103] 2) For Figure 8 The intelligent agent technology framework for artificial life forms shown is described in detail below: 2.1) System 1: Perception and Action Layer.

[0104] System 1 (corresponding to the first system described in this disclosure example) is the underlying module for direct and rapid interaction between the intelligent agent and the external environment, equivalent to the intelligent agent's "sensory and reflex nerves". It is responsible for receiving external information in real time and executing specific actions.

[0105] 2.1.1) Multi-modal Encoders: Function: This is an information processing unit, responsible for converting various raw, unprocessed information in the environment (such as a piece of text sent by the user, a picture, a piece of voice) into structured events that the agent's internal system can uniformly understand and process. These events will be marked with type (text, image or audio) and timestamp.

[0106] Implementation: It consists of a series of specialized encoding models, such as using the CLIP model to process image information, using the Whisper model to process voice information, and using a text tokenizer to process text information.

[0107] Output: The encoded structured events will be output in json format and immediately published to the agent's internal message bus, just like sensory signals entering the brain, waiting for the upper system to process. Code implementation example as follows: “{ "event_id": "composite_20250527_101520_101525", "timestamp": "2025-05-27T10:15:25.100Z", "event_type": "multi_modal ", "constituent_events": [ { "event_id": "event_20240527_101520_img_001", "timestamp": "2024-05-27T10:15:20.800Z", "event_type": "scene_detection", "modality": "image", "source": "front_camera", "encoded_data": { "detected_objects": [ { "label": "person", "attributes": { "action": "rubbing_temples", "posture": "slouching" } }, { "label": "document", "attributes": { "state": "cluttered" } }, { "label": "coffee_cup", "attributes": { "state": "empty" } } ], } }, { "event_id": "event_20250527_101525_audio_001", "timestamp": "2025-05-27T10:15:25.100Z", "event_type": "user_voice_command", "modality": "audio", "source": "microphone", "encoded_data": { "transcribed_text": "Dim the lights a bit." } } ], }".

[0108] 2.1.2) Actuator: Function: Similar to the "hands and feet" of an intelligent agent. It receives high-level commands from System 2 and translates these abstract commands into a series of atomic operations that can be executed to change the state of the environment. High-level instructions are essentially linguistic descriptions of the system's executable skills; each skill can be explicitly mapped to one or more executable operations. Example: The high-level instruction is "dance". <id>, atomic operations for the robot's motor qpos, qvel sequences, and playing the accompaniment.

[0109] Implementation: It consists of encapsulated programs that various tools and hardware bodies act upon, which can invoke search engines, manipulate browser buttons, invoke database query interfaces, or control robotic hardware, etc.

[0110] Feedback: After each action is executed, the environment provides feedback on the result. The actuator generates an external reward signal (e.g., whether the task was successful or not), which is sent directly to the hybrid reward module of system three as a direct basis for evaluating the goodness of the action.

[0111] 2.2) System 2: Deliberative Reasoning Layer.

[0112] System 2 (corresponding to the second system described in the examples of the present disclosure) is the "cerebral cortex" of the agent, where slow thinking, complex planning, and problem solving occur. System 2 receives the goals issued by system 3 and conducts deliberate decomposition and reasoning.

[0113] 2.2.1) LLM Planner (LLM Planner): Function: This is the core reasoning engine of system 2. Its task is to take the goals given by system 3, combine the information seen so far (from the event stream of system 1) and short-term memory, and conduct multi-step, detailed thinking to ultimately output a high-level plan or command to achieve the goal.

[0114] Workflow: It uses "thought chain" and other prompting engineering techniques to guide large language models (LLM) to reason step by step. For example, it first asks the LLM to analyze the goal, then lists possible steps, evaluates each step, and finally outputs an optimal solution. This solution is converted into accurate commands that can be executed by the computer by a parser.

[0115] 2.2.2) Policy Update Mechanism (PolicyUpdate): Function: This system is responsible for making the agent "smarter". It adjusts and optimizes its decision-making strategy based on the mixed total reward provided by the hybrid reward module of system 3 (combining the task score and the comprehensive evaluation of intrinsic motivation).

[0116] Implementation: Reinforcement learning algorithms (such as PPO or Neural Language RL). If a certain decision-making process brings high rewards, this mechanism will strengthen this decision-making path, making it more likely to make similar good decisions in similar situations in the future; conversely, it will weaken poor decision-making paths. This allows the agent's behavior to continuously improve and better achieve the goals set by system 3.

[0117] 2.3) System 3: Meta-cognitive Layer.

[0118] System 3 (corresponding to the third system described in the examples of the present disclosure) is the "highest decision-making center and source of self-awareness" of the agent, responsible for endowing it with life-like characteristics. It consists of a core monitor and four functional modules that work together to achieve self-supervision, long-term planning, and identity maintenance.

[0119] 2.3.1) Executive Monitor: Function: This is the highest level program of System 3, a never-ending core loop. It listens to all internal messages and coordinates three key internal processes: Thought Search: Every system cycle, especially when major tasks, scenarios, and self-set changes occur, it launches multiple "thinking threads" to explore multiple short-term goals in parallel, building a Tree-of-Thought (ToT), evaluating the potential value of each solution, and ultimately selecting the most valuable one. Specifically, when a new problem is transmitted, the executive monitor initiates the thought tree search process: by invoking multiple LLM work processes, it expands the thought path in a breadth-first manner. Each search node records a local planning solution and its corresponding value estimate . The search process terminates when any of the following conditions are met: i) The estimated value of a node exceeds the utility threshold obtained through learning, i.e. ; where represents the utility threshold.

[0120] ii) The preset search resource limit is reached.

[0121] Process Supervision: Every idea or plan that has just been generated, i.e., the leaf nodes of the thought tree TOT, will be immediately critically examined by a process supervision model, such as an LLM, to ensure its logical reasonableness and safety. From all leaf nodes, the node with the highest estimated value is selected, its content is taken as the output of the next stage, and problematic plans are eliminated or modified. That is: ; where represents the core idea / plan content generated at the moment; represents the intrinsic reward; represents the weight information; represents the content of the leaf node with the highest estimated value The next stage output content extracted from it; This represents the set of leaf nodes in the Mind Tree ToT. In the middle, find the valuation function The leaf node with the largest value Operation; This represents the set of all leaf nodes in the mind tree ToT, containing all new ideas / plans currently awaiting evaluation; Represents leaf nodes The evaluation function is used to evaluate the nodes. The quality (such as logical rationality, security, etc.) is evaluated, and the output is an estimated valuation score.

[0122] Reflection: After an interactive task is completed, conduct a post-event review, compare the expected results with the actual results, summarize successful experiences and lessons learned from failures, and extract any usable experiences into skills that can be used in the future and store them in memory.

[0123] Output: The monitor will eventually generate the next sub-goal to be executed and the adjusted intrinsic motivation weights, and issue them to System 2.

[0124] 2.3.2) Memory Module: Function: This is the agent's "autobiographical memory bank". It does not store facts in isolation, but records past experiences (including events, actions, results and the context at the time) in their entirety, forming segments of "life experiences".

[0125] Technical Implementation: The system employs a "Retrieval Enhanced Generation (RAG)" architecture. The underlying layer uses a vector database for rapid semantic-based memory retrieval, and a graph database to store the complex relationships between people, events, and objects in memory, thus achieving efficient and accurate memory retrieval. The database creates a node and relationship system centered on Events, specifically designed to represent the agent's autobiographical experiences. Emphasis is placed on dynamic and causal relationships: the focus of storage shifts from "what" to "why it happened" and "what happened afterward," which is crucial for enabling the agent to understand the past and predict the future. The system deeply integrates the relational reasoning capabilities of graphs with the semantic search capabilities of vectors. First, "relevant content" is found through vector search, then the "storyline" behind this content is understood through graph traversal, thus returning a memory unit to the system that is both detailed and contextually relevant.

[0126] 2.3.3) User Model: Function: This is the "theory of mind" capability module of the agent. It builds a dynamically updated user state summary (in JSON format) for each interacting user, which relies on a large language model to continuously infer and record the user's goals, knowledge background, emotional state, and preferences.

[0127] Effect: This makes the agent no longer mechanically responsive, but able to "read between the lines" and have empathetic, personalized interactions, thus achieving truly intelligent, humanized service.

[0128] 2.3.4) Self-Model: Function: This is the agent's self-awareness of "who am I". It explicitly records three things: Capability List: What skills do I have? (For example: I can process text, but I can't analyze pictures).

[0129] Current State: How is my "emotion" or "energy" now? (For example: I just finished a complex task and am in a high-load state).

[0130] Core Beliefs: What are my unshakable basic principles and values? (For example: "Must always prioritize helping users relieve stress").

[0131] Effect: It is the source of autonomous learning. By comparing "Capability List" and "Target Requirements", the agent can find its own shortcomings and generate learning goals (for example: "The target requires picture analysis, but I don't, so my new goal is to learn to call image recognition API").

[0132] Update Mechanism: The update of the self-model and the comparison of the capability list and the target requirements is a continuous, event-driven process. In the target generation phase (pre-event-driven), when the meta-cognitive monitor of system three needs to generate a new target, it will consult the self-model. Before each time system three actively plans and generates a new target, such a comparison will be triggered. In the reflection phase after task execution (post-event-driven), when a task (whether externally assigned or self-generated) is completed, the reflection mechanism of system three will be activated. The reflection process will evaluate the effectiveness of the entire task execution. If the task is successfully completed and a new skill is applied or an old skill is significantly improved during the process, the reflection mechanism will propose to update the self-model.

[0133] 2.3.5) Hybrid Reward Module: Function: This module is responsible for calculating the "overall score" of the agent's behavior, not only looking at whether the task is completed, but also focusing on whether the behavior is "in line with nature".

[0134] The hybrid reward includes extrinsic reward and intrinsic reward.

[0135] Extrinsic reward: objective feedback from the environment, such as task success, user likes.

[0136] Intrinsic reward: subjective motivation generated by the agent itself, including curiosity (the joy of exploring new areas), sense of control (the sense of achievement of learning new skills), consistency (the sense of security of behavior consistent with one's beliefs).

[0137] Working mode: combine intrinsic reward and extrinsic reward into a total reward according to dynamic weight. This total reward is the only basis for system two to update the strategy, so as to guide the agent not only to complete the task efficiently, but also to maintain curiosity, continuous learning and consistent behavior, like a real life.

[0138] The total reward calculation formula is as follows: ; Among them, Total reward; Intrinsic reward; Extrinsic reward; Weight coefficient.

[0139] 3) The core workflow of the agent is as follows: 3.1) Perception and event reporting: system one perceives the environment and encodes the perceived environment state data into structured environment information (such as environment events) and reports it to system three.

[0140] 3.2) Monitoring and diagnosis: meta-cognitive monitor receives events. The hybrid reward module may first detect negative feedback signals. The monitor consults the self-model and memory module and diagnoses the root cause.

[0141] 3.3) Goal generation and planning: meta-cognitive monitor starts thinking search and generates new goals. The process supervision mechanism ensures the rationality and safety of the plan.

[0142] 3.4) Instruction issuance and execution: system three issues goals and high-level plans to system two. System two performs detailed task decomposition and sends specific task commands to system one for execution.

[0143] 3.5) Reward calculation and learning: after system one converts task commands into action instructions and executes them, the hybrid reward module in system three calculates the total reward (for example: extrinsic reward is "task completion", intrinsic reward is "practicing the belief of caring for users"). The reward is used to update the strategy of system two.

[0144] 3.6) Memory and Reflection: The entire successful experience (goal, context, thought chain, result) is structured into the memory module. At the end of the cycle, the reflection process will refine the experience and update the self model.

[0145] 3.7) Cycle: The agent enters the next cognitive cycle, becomes more "mature", and can better cope with future challenges, showing self-driven learning and exploration of life-like behavior.

[0146] Through the above architecture and process, the invention successfully transforms a static, task-driven agent into a "life-like artificial life" system with continuous identity, self-driven learning and exploration.

[0147] In the embodiments of the present disclosure, by introducing the system ternary cognitive layer and its supporting mechanism, the agent system has brought many breakthroughs and benefits, as follows: ① Achieve long-term identity consistency and behavior coherence.

[0148] Technical means: Through the memory module, the RAG (Retrieval Augmentation Generation) technology architecture is adopted, combined with vector database and graph database, to structure the storage of experience events with complete context.

[0149] Advantages: The agent can maintain a stable "narrative self" across different task sessions, avoiding the problem of losing context due to session reset in traditional agents, providing true long-term companionship and service capability.

[0150] ② Realize autonomous goal generation and self-driven evolution.

[0151] Technical means: The self model continuously maintains and updates the agent's ability list, and compares it with the target requirements issued by the system in real time.

[0152] Advantages: The agent can autonomously identify knowledge and skill gaps and generate new learning goals without relying on external task scheduling, realizing a fundamental change from "passive execution" to "active learning", and possessing self-evolution ability like a life-like body.

[0153] ③ Improve the reliability and safety of complex decision-making.

[0154] Technical means: Through the process supervision mechanism in the meta-cognition monitor, each reasoning step generated by the system two is audited, and a special process supervision model checks the logical consistency and safety.

[0155] Advantages: Errors or harmful reasoning branches can be found and pruned in time before plan execution, greatly reducing the risk of wrong decisions and safety hazards, providing reliable protection for the autonomous operation of the agent.

[0156] IV. Effect: significantly reduces the cost of computing for long-term operation.

[0157] Technical means: store successful inference trajectories through the memory module, and directly retrieve and reuse them when encountering similar situations (forward learning) to avoid repeated calls to large models for complex reasoning.

[0158] Advantages: significantly reduces the number of calls to large language models (LLM) for completing similar tasks, effectively controls cloud computing costs, and makes large-scale, long-term deployment of intelligent agents economically feasible.

[0159] V. Achieve personalized and empathetic social interaction.

[0160] Technical means: continuously track and infer the beliefs, goals, and emotional states of interaction objects through user models.

[0161] Advantages: enables intelligent agents to customize communication strategies and behaviors based on deep understanding of users, providing truly personalized and empathetic service experiences, and enhancing the naturalness of interactions and user satisfaction.

[0162] VI. Achieve transparent and auditable decision-making process.

[0163] Technical means: generate complete "thought trees" through the meta-cognitive monitor's thought search mechanism, and persistently store all inference trajectories and reflection logs.

[0164] Advantages: provides a clear traceability path for each decision, making the behavior logic of intelligent agents transparent, auditable, and explainable to designers and regulators, meeting the requirements of artificial intelligence ethics and safety alignment.

[0165] Those skilled in the art of the technical field to which the present disclosure belongs will understand that various aspects of the present disclosure can be implemented in the form of a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, which can be collectively referred to as "circuitry", "modules", or "systems".

[0166] Based on the same inventive concept, the present disclosure also provides an electronic device, which includes a processor and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the data processing method of the intelligent agent system of any one of the above embodiments by executing the executable instructions. Since the problem-solving principle of this electronic device embodiment is similar to that of the above method embodiments, the implementation of this electronic device embodiment can be referred to the implementation of the above method embodiments, and the repeated parts will not be described again.

[0167] The following refers to Figure 9 The electronic device 900 according to this implementation of the present disclosure is described. Figure 9 The electronic device 900 shown is merely one example. It should be appreciated that the functions and the use of the electronic device 900 described herein are not limited by this example.

[0168] As shown in Figure 9 The electronic device 900 is in the form of a general computing device. Components of the electronic device 900 can include, but are not limited to, the at least one processing unit 910 described above, the at least one storage unit 920 described above, and a bus 930 that connects the various system components, including the storage unit 920 and the processing unit 910.

[0169] The storage unit stores program code that can be executed by the processing unit 910, so that the processing unit 910 performs the steps described in the "Exemplary Method" section above according to various exemplary implementations of the present disclosure. For example, the processing unit 910 can perform the following steps of the above method embodiments: obtaining a target task plan to be executed; performing task reasoning decomposition on the target task plan based on an agent policy model of a large language model to generate an action command executable by an agent; updating the agent policy model according to hybrid reward information in response to the agent executing each action command, the hybrid reward information being a fusion result of task reward information and system intrinsic reward information; converting perceived environment state data into structured environment perception information recognizable and / or processable by an agent system internally; updating an agent autonomous model for the agent system to execute an autonomous learning task plan according to historical interaction information between the agent and the environment, and the structured environment perception information, reasoning context information, agent action information, and task execution result in the process of executing the target task plan.

[0170] The storage unit 920 can include a readable medium in the form of volatile storage such as random access memory (RAM) 9201 and / or cache memory 9202, and can further include a non-volatile storage such as read-only memory (ROM) 9203.

[0171] The storage unit 920 can further include program / utility 9204 having a set of program modules 9205, including but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which can include implementation of a network environment, alone or in some combination.

[0172] The bus 930 can be representative of one or more of several types of bus structures, including a storage unit bus or storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0173] The electronic device 900 can also communicate with one or more external devices 940 such as a keyboard or pointing devices, a Bluetooth device, or a disk drive. These and other input / output devices are connected to the electronic device 900 through input / output interfaces 950 that are coupled via the bus 930. Still yet, the electronic device 900 can communicate with one or more networks, such as one or more areas networks (LANs), wide area networks (WANs), or the Internet, through a network adapter 960 that is connected to the bus 930. As depicted, the network adapter 960 communicates with the other components of the electronic device 900 through the bus 930. It should be appreciated that the network adapter 960 and / or the input / output interfaces 950 can be implemented with one or more types of technologies that are now existing or are later developed and that the network adapter 960 and / or the input / output interfaces 950 can be implemented using hardware, software, or a combination thereof.

[0174] Those skilled in the art will readily understand that the example embodiments described herein can be implemented by software and / or by software in combination with the necessary hardware. Thus, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash disk, a mobile hard disk, or the like) or a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to perform the methods according to the embodiments of the present disclosure.

[0175] Based on the same inventive concept, the present disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method of the agent system according to any one of the above embodiments. Since the computer-readable storage medium embodiment solves problems in the same principle as the above method embodiments, the implementation of the computer-readable storage medium embodiment can be referred to the implementation of the above method embodiments, and the repeated parts will not be described herein.

[0176] More specific examples of the computer-readable storage medium in the present disclosure can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0177] In this disclosure, a computer readable storage medium can include a data signal transporting or broadcasting computer readable program code embodied in the data signal. The data signal can also be transmitted over a network including the Internet. Additionally, the computer readable storage medium can also be any tangible storage medium which can be used to store and / or carry the program codes.

[0178] Optionally, program code embodied on a computer readable storage medium can also be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0179] In an implementation, the program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ and the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider. The present disclosure can also be implemented as a computer program product, which can include a computer readable storage medium having computer readable program code embodied in said medium.

[0180] Based on the same inventive concept, the present disclosure also provides a computer program product, which includes a computer program product, including: a computer program or instructions, which, when executed by a processor, implements the data processing method of the agent system of any one of the above method embodiments. Since the problem-solving principle of this computer program product embodiment is similar to that of the above method embodiments, the implementation of this computer program product embodiment can be referred to the implementation of the above method embodiments, and repeated parts will not be described here.

[0181] It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, such division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into a plurality of modules or units for embodiment.

[0182] Moreover, although individual steps of the methods in the present disclosure are described in a particular order in the drawings, this is not required or implied, nor is it necessary to perform all of the steps shown to achieve the desired result. Additionally or alternatively, certain steps can be omitted, combined into a single step, performed in a different order, broken down into multiple steps, and / or the like.

[0183] Those skilled in the art will readily understand that the example embodiments described herein can be implemented by software and / or by hardware coupled with software, as described above. Thus, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash disk, a mobile hard disk, or the like) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to perform the methods according to the embodiments of the present disclosure.

[0184] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure following the general principles thereof and including such departures from the present disclosure that come within known use or custom in the art to which the present disclosure pertains. The specification and examples are to be regarded as illustrative only, and the true scope and spirit of the present disclosure are indicated by the appended claims.< / id>

Claims

1. An agent system, characterized by Comprise: A first system interacting with an external environment, a second system deploying an agent policy model, and a third system deploying an agent autonomous model; Wherein, the first system is used to convert the perceived environment state data into structured environment perception information recognizable and / or processable inside the agent system, transmit to the third system, and execute the action command issued by the second system; The second system is used to perform task reasoning decomposition on the target task plan issued by the third system based on the agent policy model of the large language model, generate the action command executable by the agent, and issue it to the first system; The third system is used to generate a target task plan and a hybrid reward information for updating the agent policy model, issue it to the second system, and update the agent autonomous model according to the historical interaction information between the agent and the environment, the structured environment perception information, the reasoning context information, the agent action information, and the task execution result in the target task plan execution process, the agent autonomous model is used for the agent system to execute autonomous learning task plan, and the hybrid reward information is the fusion result of task reward information and intrinsic reward information inside the system.

2. The agent system of claim 1, wherein, The first system comprises a multi-modal perception encoder and an action executor; Wherein, the multi-modal perception encoder is used to convert the perceived multi-modal environment state data into structured environment perception information recognizable and / or processable inside the agent system; The action executor is used to execute the action command issued by the second system.

3. The agent system of claim 1, wherein, The second system comprises a large language model planner and a policy update module; Wherein, the large language model planner is used to perform task reasoning decomposition on the target task plan issued by the third system based on the agent policy model of the large language model, and generate the action command executable by the agent; The policy update module is used to update the agent policy model according to the hybrid reward information generated by the third system.

4. The agent system of claim 1, wherein, The third system comprises a meta-cognition monitor, a memory module, a user model, an agent autonomous model, and a hybrid reward module; Wherein, the meta-cognition monitor is used to generate a target task plan and issue it to the second system; The memory module is used to structure the storage of the structured environment perception information, the reasoning context information, the agent action information, and the task execution result in the target task plan execution process based on the vector database and / or the graph database; The user model is used to establish dynamic updated user state information for each user; The agent autonomous model is used to compare the capability information of the agent system with the task demand in response to the event trigger of generating a target task plan or completing a target task plan, and generate an autonomous learning task plan of the agent system according to the comparison result; The hybrid reward module is used to generate the hybrid reward information for updating the agent policy model according to the task reward information provided by the second system and the intrinsic reward information generated inside the agent system.

5. A data processing method of an agent system, characterized by, Comprise: Obtain a target task plan to be executed; The agent policy model based on the large language model performs task reasoning decomposition on the target task plan, and generates action commands executable by the agent; In response to the agent executing each action command, the agent policy model is updated according to mixed reward information, which is a fusion result of task reward information and system intrinsic reward information; The perceived environment state data is converted into structured environment perception information recognizable and / or processable by the agent system internally; The agent autonomous model is updated according to historical interaction information between the agent and the environment, structured environment perception information, reasoning context information, agent action information, and task execution results in the target task plan execution process, and the agent autonomous model is used for the agent system to execute autonomous learning task plans.

6. The data processing method of an agent system according to claim 5, wherein, The method further comprises: The structured environment perception information, reasoning context information, agent action information, and task execution results in the target task plan execution process are stored in a structured manner.

7. The data processing method of an agent system according to claim 5, wherein, After the agent policy model based on the large language model performs task reasoning decomposition on the target task plan and generates action commands executable by the agent, the method further comprises: Each action command output by the agent policy model is verified based on a pre-trained process supervision model; It is determined whether to execute the corresponding action command according to the verification result.

8. An electronic device, comprising: It comprises: a processor; and a memory for storing executable instructions of the processor; The processor is configured to execute the executable instructions to perform the data processing method of the agent system according to any one of claims 5 to 7. The computer program is executed by the processor to implement the data processing method of the agent system according to any one of claims 5 to 7.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program or instructions are executed by the processor to implement the data processing method of the agent system according to any one of claims 5 to 7.

10. A computer program product, comprising: ​

Images