Method, device, and system for policy optimization based on multiple value functions

Abstract

An embodiment of the present disclosure may provide a system comprising: one or more processors; and one or more memories that collectively store instructions that, when executed by the one or more processors, cause the system to perform operations. In an embodiment, the operations may comprise the operations of: learning a first value function by performing offline reinforcement learning on the basis of offline data; learning a second value function by performing online pre-training; and performing online reinforcement learning by using a third value function generated on the basis of the first value function and the second value function.

Description

Multiple value function-based policy optimization methods, devices, and systems

[0001] The present disclosure relates to a method, apparatus, and system for optimizing a policy using multiple value functions. More specifically, the present disclosure relates to a method, apparatus, and system for optimizing a policy using multiple value functions when using offline learning and online learning.

[0002] Reinforcement learning is an artificial intelligence learning method in which an agent learns how to make decisions by interacting with its environment, primarily used in applications such as robot control and autonomous driving. Reinforcement learning can encompass both online and offline reinforcement learning. Online reinforcement learning is a learning method where the agent directly interacts with the environment to collect data. In contrast, offline reinforcement learning is a method where the agent does not directly interact with the environment; instead, a separate behavioral algorithm exists to learn policies based on pre-collected fixed data without interaction with the environment. While offline reinforcement learning offers the advantage of enabling learning without risk in real-world environments, such as in robots and autonomous driving, it suffers from a decline in reasoning ability in situations outside of the pre-collected fixed data. Consequently, research is currently being conducted on methods that utilize both online and offline reinforcement learning for training.

[0003] One embodiment of the present disclosure aims to solve the problem of inaccuracy in value function estimation during the process of performing online reinforcement learning after offline reinforcement learning.

[0004] A system according to one embodiment of the present disclosure comprises one or more processors; and one or more memories that collectively store instructions that cause the system to perform operations when executed by the one or more processors, wherein the operations may include: an operation of learning a first value function by performing offline reinforcement learning based on offline data; an operation of learning a second value function by performing online pre-learning; and an operation of performing online reinforcement learning using a third value function generated based on the first value function and the second value function.

[0005] In one embodiment, the third value function may be generated by combining the first value function and the second value function as a weighted sum.

[0006] In one embodiment, the weights of the first value function and the second value function may be updated according to the degree of online reinforcement learning performance.

[0007] In one embodiment, the weights of the first value function and the second value function can be adjusted according to the distribution difference indicator between the offline data distribution and the online data distribution.

[0008] In one embodiment, the third value function has a weight of the first value function at a first time point that is greater than the weight of the first value function at a second time point, and a weight of the second value function at a first time point that is smaller than the weight of the second value function at a second time point, and the first time point may be a time point earlier than the second time point.

[0009] In one embodiment, the offline reinforcement learning is learned based on the offline data and the offline policy, and the operation of learning a second value function by performing the online pre-learning may include: an operation of acquiring online transition data by performing limited interaction with the environment according to the offline policy while maintaining the offline policy; and an operation of learning a second value function by performing the online pre-learning based on the offline data and the online transition data.

[0010] In one embodiment, the offline reinforcement learning is learned based on the offline data and the offline policy, and the operation of learning a second value function by performing the online pre-learning may include: an operation of acquiring online transition data by performing limited interaction with the environment according to the offline policy while maintaining the offline policy and the first value function; and an operation of learning a second value function by performing the online pre-learning based on the offline data and the online transition data while maintaining the offline policy and the first value function.

[0011] A method according to one embodiment of the present disclosure may include: learning a first value function by performing offline reinforcement learning based on offline data; learning a second value function by performing online pre-learning; and performing online reinforcement learning using a third value function generated based on the first value function and the second value function.

[0012] In one embodiment, the third value function may be generated by combining the first value function and the second value function as a weighted sum.

[0013] In one embodiment, the weights of the first value function and the second value function may be updated according to the degree of online reinforcement learning performance.

[0014] In one embodiment, the weights of the first value function and the second value function can be adjusted according to the distribution difference indicator between the offline data distribution and the online data distribution.

[0015] In one embodiment, the third value function has a weight of the first value function at a first time point that is greater than the weight of the first value function at a second time point, and a weight of the second value function at a first time point that is smaller than the weight of the second value function at a second time point, and the first time point may be a time point earlier than the second time point.

[0016] In one embodiment, the offline reinforcement learning is learned based on the offline data and the offline policy, and the operation of learning a second value function by performing the online pre-learning may include: an operation of acquiring online transition data by performing limited interaction with the environment according to the offline policy while maintaining the offline policy; and an operation of learning a second value function by performing the online pre-learning based on the offline data and the online transition data.

[0017] In one embodiment, the offline reinforcement learning is learned based on the offline data and the offline policy, and the operation of learning a second value function by performing the online pre-learning may include: an operation of acquiring online transition data by performing limited interaction with the environment according to the offline policy while maintaining the offline policy and the first value function; and an operation of learning a second value function by performing the online pre-learning based on the offline data and the online transition data while maintaining the offline policy and the first value function.

[0018] One embodiment of the present disclosure includes a program stored on a recording medium to execute a method according to one embodiment of the present disclosure on a computer.

[0019] One embodiment of the present disclosure includes a computer-readable recording medium having a program for executing a method according to one embodiment of the present disclosure on a computer.

[0020] One embodiment of the present disclosure includes a computer-readable recording medium that records a database used in one embodiment of the present disclosure.

[0021] According to one embodiment of the present disclosure, a policy and value function learned in the offline phase are frozen (maintained) during online pre-training, and a second value function adaptive to the online distribution is separately pre-trained, thereby mitigating value estimation distortions caused by distribution shift.

[0022] In addition, according to one embodiment of the present disclosure, the method has a smaller scope of implementation change compared to existing offline-only or purely online methods and can be easily integrated into existing pipelines, resulting in high cost-effectiveness in both research and industrial environments.

[0023] FIG. 1 is a drawing showing a sequential multi-agent according to one embodiment of the present disclosure.

[0024] FIG. 2 is a drawing showing a supervisory agent according to one embodiment of the present disclosure.

[0025] FIG. 3 is a diagram showing a hierarchical agent system according to one embodiment of the present disclosure.

[0026] FIG. 4 is a drawing showing a multi-agent discussion type system according to one embodiment of the present disclosure.

[0027] FIG. 5 is a diagram showing a Mixture-of-AI Agents system according to one embodiment of the present disclosure.

[0028] FIG. 6 is a drawing showing a ReAct agent system according to one embodiment of the present disclosure.

[0029] FIG. 7 is a drawing showing a CodeAct agent system according to one embodiment of the present disclosure.

[0030] FIG. 8 is a drawing showing a modern tool usage agent system according to one embodiment of the present disclosure.

[0031] FIG. 9 is a drawing showing a self-reflective agent system according to one embodiment of the present disclosure.

[0032] FIG. 10 is a drawing showing a multi-agent workflow system according to one embodiment of the present disclosure.

[0033] FIG. 11 is a drawing showing an Agentic RAG (Retrieval-Augmented Generation) system according to one embodiment of the present disclosure.

[0034] FIG. 12 is a drawing showing a Multi-Agent Debate (MAD) system according to one embodiment of the present disclosure.

[0035] FIG. 13 is a diagram showing an A2A (Agent2Agent) protocol system according to one embodiment of the present disclosure.

[0036] FIG. 14 is a drawing showing an Agentic RAG system according to one embodiment of the present disclosure.

[0037] FIG. 15 is a schematic diagram of an AI agent system according to one embodiment of the present disclosure.

[0038] FIG. 16 is a schematic diagram of an LLM chatbot according to one embodiment of the present disclosure.

[0039] FIG. 17 is a schematic diagram of a Robotic Process Automation (RPA) system according to one embodiment of the present disclosure.

[0040] FIG. 18 is a schematic diagram of a RAG (Retrieval-Augmented Generation) system according to one embodiment of the present disclosure.

[0041] FIG. 19 is a schematic diagram of a Learning-Augmented Mechanism (LAM) according to one embodiment of the present disclosure.

[0042] FIG. 20 is a diagram showing an AI agent memory structure according to one embodiment of the present disclosure.

[0043] FIG. 21 is a drawing showing a GPT (General Pretrained Transformer) model according to one embodiment of the present disclosure.

[0044] FIG. 22 is a drawing showing a Mixture of Experts (MoE) model according to one embodiment of the present disclosure.

[0045] FIG. 23 is a drawing showing a Large Reasoning Model (LRM) according to one embodiment of the present disclosure.

[0046] FIG. 24 is a drawing showing a Vision Language Model (VLM) according to one embodiment of the present disclosure.

[0047] FIG. 25 is a drawing showing a Small Language Model (SLM) according to one embodiment of the present disclosure.

[0048] FIG. 26 is a drawing showing a Large Action Model (LAM) according to one embodiment of the present disclosure.

[0049] FIG. 27 is a drawing showing a Hierarchical Reasoning Model (HRM) according to one embodiment of the present disclosure.

[0050] FIG. 28 is a drawing showing a ToolFormer (Tools-trained Model) according to one embodiment of the present disclosure.

[0051] FIGS. 29 to 34 are drawings illustrating vulnerabilities of an MCP according to one embodiment of the present disclosure.

[0052] FIG. 35 is a diagram illustrating a context engineering structure in an AI agent system according to one embodiment of the present disclosure.

[0053] Figure 36 is a diagram illustrating an offline-online learning method.

[0054] FIG. 37 is a drawing illustrating an offline-online learning method according to one embodiment of the present disclosure.

[0055] FIG. 38 is a block diagram of a system according to one embodiment of the present disclosure.

[0056] FIG. 39 is a flowchart illustrating an offline-online learning method according to one embodiment of the present disclosure.

[0057] Figure 40 is for an exemplary on-premises full-stack structure.

[0058] To clarify the technical concept of the present disclosure, embodiments of the present disclosure will be described in detail with reference to the attached drawings. In describing the present disclosure, detailed descriptions of related known functions or components will be omitted if it is determined that such detailed descriptions would unnecessarily obscure the essence of the present disclosure. Components having substantially the same functional configuration among the drawings have been assigned the same reference numerals and symbols as much as possible, even if they are shown in different drawings. For convenience of explanation, devices and methods will be described together where necessary. Each operation of the present disclosure does not necessarily have to be performed in the order described and may be performed in parallel, selectively, or individually.

[0059] The terms used in the embodiments of this disclosure have been selected to be as widely used and general as possible, taking into account the functions of this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been selected at the applicant's discretion, and in such cases, their meanings will be described in detail in the description of the relevant embodiments. Therefore, terms used in this specification should be defined not merely by their names, but based on their meanings and the overall content of this disclosure.

[0060] Throughout this disclosure, singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms such as “comprising” or “having” are intended to specify the presence of features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof. That is, throughout this disclosure, when a part is described as “comprising” a certain component, it means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0061] Expressions such as "at least one" modify the entire list of components and do not modify the components of the list individually. For example, "at least one of A, B, and C" and "at least one of A, B, or C" refer to only A, only B, only C, both A and B, both B and C, both A and C, all of A, B, and C, or any combination thereof.

[0062] Additionally, terms such as “...part,” “...module,” etc., as described in this disclosure refer to a unit that processes at least one function or operation, and may be implemented in hardware or software, or a combination of hardware and software.

[0063] Throughout the entire disclosure, when a part is described as being “connected” to another part, this includes not only cases where they are “directly connected” but also cases where they are “electrically connected” with other elements interposed between them. Furthermore, when a part is described as “comprising” a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0064] As used throughout this disclosure, the expression “configured to” may be replaced, depending on the context, with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” may not necessarily mean only “specifically designed to” in hardware. Instead, in some situations, the expression “system configured to” may mean that the system is “capable of” together with other devices or components. For example, the phrase “a processor configured (or set) to perform A, B, and C” may mean a dedicated processor for performing said operations (e.g., an embedded processor), or a generic-purpose processor (e.g., a CPU or an application processor) capable of performing said operations by executing one or more software programs stored in memory.

[0065] In one embodiment, agents performing online reinforcement learning can learn strategies for making optimal decisions by interacting in real-time with the environment. However, since such interaction with the environment involves learning from experience data collected by the agent, the cost of data collection may be substantial or the agent may be exposed to significant risk. To mitigate these drawbacks, offline reinforcement learning, which derives optimal policies from pre-collected data, is being researched.

[0066] AI agents are computer system implementations of human decision-making and behavioral processes, and they have established themselves as a core component of current AI systems. The basic structure of an AI agent follows a cyclical pattern of 'Perception → Knowledge Base → Reasoning & Planning → Action & Actuation → Learning Loop.' Through this structure, AI agents perceive their environment, make decisions based on collected information, execute the results, and learn from those results to continuously improve.

[0067] The agent's workflow may include the types in [Table 1] as follows.

[0068] [Correction pursuant to Rule 91 Nov. 18, 2025]

[0069] [Correction pursuant to Rule 91 Nov. 18, 2025]

[0070] [Correction pursuant to Rule 91 Nov. 18, 2025]

[0071] [Correction pursuant to Rule 91 Nov. 18, 2025]

[0072] In one embodiment, most of the structure may consist of a hierarchical structure of "Input → Planning / Evaluation / Branching → Output". Additionally, due to the agentic nature, complex problem solving may be possible through patterns such as iteration, parallelism, and collaboration, rather than a single LLM call. Furthermore, since each structure is designed to suit a specific business purpose, selecting the optimal structure according to the objective is important.

[0073] For example, a workflow like [Table 2] can be recommended depending on the purpose of the work.

[0074] [Table 2]

[0075]

[0076] FIG. 1 is a drawing showing a sequential multi-agent according to one embodiment of the present disclosure.

[0077] Referring to FIG. 1, a sequential multi-agent may include a user agent, a write agent, a style agent, etc. In one embodiment, the sequential agents communicate sequentially and can perform a single task in order. Additionally, each agent can receive the result of the previous step and perform the next task. For example, the user agent may obtain user input from the user to generate a first processing result, the write agent may generate a second processing result including the written text by generating the first processing result from the user agent, and the style agent may generate a third processing result by receiving the second processing result from the write agent and applying a style. Accordingly, the third processing result may finally be output. Such a sequential multi-agent has a linear flow and is suitable for processing a single task. For example, the sequential multi-agent can be used in the field of creative writing.

[0078] FIG. 2 is a drawing showing a supervisory agent according to one embodiment of the present disclosure.

[0079] Referring to FIG. 2, a supervised agent may refer to an agent in which a centrally located Supervisory Language Model (LLM) coordinates the entire process. In one embodiment, the supervised agent may direct necessary tasks to appropriate agents and synthesize results according to a user's request. That is, the supervised agent can manage communication between agents. Accordingly, the supervised agent can enable flexible task distribution. For example, the supervised agent may request research from a research agent and request calculations from a mathematics agent. Such a supervised agent can be used in fields such as deep research.

[0080] FIG. 3 is a diagram showing a hierarchical agent system according to one embodiment of the present disclosure.

[0081] Referring to FIG. 3, a hierarchical agent system can refer to a system in which a meta-agent controls and coordinates lower-level agents. For example, a meta-agent can obtain user input from a user, request tasks from research agents, data analysis agents, etc., receive task results from each agent, and generate outputs. This hierarchical agent system has a hierarchical control structure, allowing tasks to be divided and managed in a more complex manner. In other words, a meta-agent acting as an intermediate manager can be utilized. This hierarchical agent system is suitable for complex systems or coding, and can be used as a coding agent, etc.

[0082] FIG. 4 is a drawing showing a multi-agent discussion type system according to one embodiment of the present disclosure.

[0083] Referring to Fig. 4, a multi-agent discussion system is a system in which multiple agents present different opinions and select the most appropriate result by voting or evaluating it, thereby deriving the optimal solution based on discussion. That is, multiple agents perform discussion and evaluation based on user input and can output the optimal answer among them. Such a multi-agent discussion system can make the best choice by comparing various perspectives in a competitive structure and can be used in fields such as world simulation.

[0084] FIG. 5 is a diagram showing a Mixture-of-AI Agents system according to one embodiment of the present disclosure.

[0085] Referring to FIG. 5, in a hybrid AI agent system, multiple agents perform parallel processing layer by layer, and an aggregator can integrate the results at the end. For example, a first agent and a second agent may perform a process in parallel at the first layer, and then the first agent and the second agent may perform a process in parallel at the second layer, and an aggregator may synthesize this to generate an output. In one embodiment, the hybrid AI agent system may use a multi-stage approach for complex problems. The hybrid AI agent system has a hierarchical and parallel structure and is characterized by the distribution and combination of expertise, so it can be used for medical research, etc.

[0086] FIG. 6 is a drawing showing a ReAct agent system according to one embodiment of the present disclosure.

[0087] Referring to FIG. 6, ReAct is a compound word of "Reason + Act," and a ReAct agent can refer to an agent that solves problems by repeating reasoning (Reason) and action (Act). For example, if a user asks, "What is the weather like in New York these days?", the LLM can interpret the meaning of the question, search for the current weather in New York through a search engine's search tool, summarize the results, and deliver them back to the user. Such a ReAct agent can be used in AI chatbots, etc. According to one embodiment, the ReAct agent has excellent tool usage capabilities and can generate more accurate responses through the repetition of reasoning and action.

[0088] FIG. 7 is a drawing showing a CodeAct agent system according to one embodiment of the present disclosure.

[0089] Referring to Fig. 7, the CodeAct agent can handle more flexible and complex logic by executing Python code instead of JSON. For example, when it receives input from a user such as "Analyze sales data for the last 3 months," the LLM analyzes the request, and Pandas can be used for Python code. Subsequently, tasks such as loading CSV files, calculating statistics, generating graphs, and creating summary reports can be performed. Because the CodeAct agent is code-based, it is strong in handling complex calculations and logic, and the LLM can directly program and execute it.

[0090] FIG. 8 is a drawing showing a modern tool usage agent system according to one embodiment of the present disclosure.

[0091] Referring to Fig. 8, the agent using modern tools can easily utilize various SaaS tools or APIs (e.g., AWS, Brave search, etc.) through a Multi-Channel Processing (MCP) server. For example, when the agent receives a text request from a user saying "Stop my AWS EC2 instance," it can call the AWS API through the MCP server and return a message indicating successful stop. The agent using modern tools can be used in developer IDE-integrated AI, etc., and has the advantage of enabling tool control with almost no code and facilitating easy integration with various cloud or web functions.

[0092] FIG. 9 is a drawing showing a self-reflective agent system according to one embodiment of the present disclosure.

[0093] Referring to Fig. 9, the self-reflective agent system can refer to a metacognitive mechanism in which the LLM evaluates and modifies its own responses. For example, it could be a system in which, upon receiving input from a user such as "Write a cover letter that fits my resume," the LLM generates a draft, the Critique LLM checks for logical and contextual errors, and after iterative modification and improvement, generates a final output. The self-reflective agent system can automatically improve quality and incrementally enhance performance through a feedback loop.

[0094] FIG. 10 is a drawing showing a multi-agent workflow system according to one embodiment of the present disclosure.

[0095] Referring to Fig. 10, a multi-agent workflow system can refer to a system in which multiple specialized agents cooperate to perform a single task. For example, upon receiving input from a user such as "Please write a startup market research report," the first agent can collect the latest market trends, the second agent can analyze competitors, and the third agent can summarize investment trends. An aggregator can synthesize the information generated by the first, second, and third agents to generate a report. The multi-agent workflow system can improve accuracy through a cooperative structure and distribute and process complex tasks.

[0096] FIG. 11 is a drawing showing an Agentic RAG (Retrieval-Augmented Generation) system according to one embodiment of the present disclosure.

[0097] Referring to Fig. 11, an Agentic RAG (Retrieval-Augmented Generation) system can be described as a system in which AI retrieves information from external databases or search engines in real time and generates a response based on it. For example, when a query such as "What are the major issues of the 2024 US presidential election?" is obtained from a user, the AI ​​performs a vector DB / web search (Google, News, etc.), extracts relevant articles and summaries, and can write an explanation using that information. Such an Agentic RAG (Retrieval-Augmented Generation) system can provide the latest information and generate an accurate response that fits the context.

[0098] FIG. 12 is a drawing showing a Multi-Agent Debate (MAD) system according to one embodiment of the present disclosure.

[0099] Referring to FIG. 12, a Multi-Agent Debate (MAD) system may refer to a system in which multiple small language models derive an answer through discussion. An aggregator, such as an aggregating LLM, can receive a query from a user and combine the opinions of multiple small language models to generate a final answer. In one embodiment, when a user inputs a question, the aggregator generates an initial answer, and multiple Small Language Models (SLMs) can present different answers and refute each other. For example, when the first SLM presents an answer such as "This is the answer," the second SLM presents "No, this is the answer. I verified it this way," and the third SLM presents "I think it is almost correct, but there are these points," the aggregator can make an intermediate judgment based on the discussion content. Based on this first discussion and previous judgment, a more refined second discussion can proceed. Some models can verify facts by utilizing tools (search, vector DB, etc.). After repeating these discussions, the aggregator can determine the most appropriate response as the final verdict and deliver it to the user. In one embodiment, various SLMs can participate in discussion and verification with each other.

[0100] FIG. 13 is a diagram showing an A2A (Agent2Agent) protocol system according to one embodiment of the present disclosure.

[0101] Referring to Fig. 13, the A2A (Agent2Agent) protocol enables communication without sharing data with each other, allows for task distribution and negotiation among multiple agents, and enables each agent to maintain shared context and state information. In the example of Fig. 13, the first AI agent (AI Agent 1) can primarily perform local-based file or search tasks and can connect to various MCP servers via the MCP Protocol. In the example of Fig. 13, the second AI agent (AI Agent 2) can primarily handle cloud and communication tasks and can connect to various MCP servers via the MCP Protocol. MCP allows for communication by separating each function (file access, search, cloud, etc.) into separate servers. Furthermore, A2A has the advantage of high security because it enables direct communication between agents without sharing data. Each agent can operate independently using its own language model, framework, and database.

[0102] FIG. 14 is a drawing showing an Agentic RAG system according to one embodiment of the present disclosure.

[0103] Referring to FIG. 14, the Agentic RAG system may refer to a Retrieval-Augmented Generation (RAG) system that extracts data from a website, stores it in a vector database, searches for similar information in response to a user query, and generates a response through a Large Language Model (LLM). In particular, this structure can support advanced question-answering by including agent functions (Memory, Tools, Planning, etc.). In one embodiment, the Agentic RAG system may include a data extraction step, a search step, and a generation step. The data extraction step may include a step of extracting data from a designated website (e.g., GitHub, Hacker News, etc.) or web content (website content in various formats such as text, images, audio, video, etc.). Additionally, the data extraction step may include a preprocessing and storage step. The preprocessing and storage step may include a step of extracting text and metadata from the content, a step of chunking the text into small units, a step of vectorizing each piece through an embedding model, and a step of storing the vectorized data in a vector database (Vector DB).

[0104] In one embodiment, the search step of the Agentic RAG pipeline may include the steps of: inputting a user query; performing embedding after query rewriting; searching for similarity in a vector database; configuring the search results into context; and ranking based on relevance for the context.

[0105] In one embodiment, the generation step of the Agentic RAG pipeline may include: a step of generating an input by combining a user query and a retrieved context; a step in which a large language model (LLM) generates a response—including agentic elements such as memory functions, tool calls, and planning functions; and a step of providing the generated response to the user.

[0106] In one embodiment, website content is collected and textified, text segmentation and embedding are performed, stored in a vector database, user query embeddings and similar content are searched, relevant context is constructed and LLM input is expanded, and an LLM-based response can be generated and delivered to the user.

[0107] In one embodiment, the Agentic RAG system is a search-based generation (RAG) system capable of generating precise and contextually relevant answers based on its architecture, and capable of handling complex queries through memory storage, calling external tools, and planning. It also supports various forms of content such as images, audio, and video in addition to text, enabling multimodal input processing, and can improve search accuracy by including query rewriting and ranking functions.

[0108] In one embodiment, the Agentic RAG system can collect real-time information from a website and utilize it for question and answer, and can be applied to various services such as technical support, search engine enhancement, and personal assistant services, and can be utilized for complex task automation through Agentic components.

[0109] FIG. 15 is a schematic diagram of an AI agent system according to one embodiment of the present disclosure.

[0110] Referring to FIG. 15, based on system prompts and user prompts, the AI ​​system can generate a final response by performing actions such as formulating a plan, calling a tool, storing it in memory, and collecting feedback. This AI agent system features an autonomous planning and execution structure and can dynamically select and execute tools. In addition, the AI ​​agent system enables continuous learning and improvement through a feedback loop, so it can be used for complex multi-step tasks, business automation, research, etc.

[0111] An agent system according to one embodiment of the present disclosure may be an advanced agent system that goes beyond a simple input-output structure by incorporating complex functions such as memory, reasoning, tool integration, and planning.

[0112] FIG. 16 is a schematic diagram of an LLM chatbot according to one embodiment of the present disclosure.

[0113] Referring to Fig. 16, a Large Language Model (LM) can take user input as input and output a response. This system consists of a single LLM call and does not involve external tool calls or complex workflows, making it suitable for simple question and answer, FAQ, and customer service. However, this system may lack the ability to maintain context or perform repetitive tasks.

[0114] FIG. 17 is a schematic diagram of a Robotic Process Automation (RPA) system according to one embodiment of the present disclosure.

[0115] Referring to Fig. 17, the tool can be executed according to predefined fixed rules using user input. In this case, the LLM may be limited to a secondary role. Such an RPA system is suitable for repetitive automation and simple back-office tasks, but it may have limitations in flexible flow control or advanced decision-making.

[0116] FIG. 18 is a schematic diagram of a RAG (Retrieval-Augmented Generation) system according to one embodiment of the present disclosure.

[0117] Referring to Fig. 18, user input is embedded and converted into a vector DB, and relevant information is searched to create an augmented prompt that is processed via LLM, and a final response can be output. This RAG system is suitable for fields such as generating correct answers using external knowledge (documents, etc.), improving search-based LLM accuracy, QA systems, document search, and summarization, but there are limitations in processing information outside the scope of the searched information.

[0118] FIG. 19 is a schematic diagram of a Learning-Augmented Mechanism (LAM) according to one embodiment of the present disclosure.

[0119] Referring to Fig. 19, the LAM system can perform a process in which user input is processed by an LLM based on training data, and a task is executed by running a tool to output a response. The LAM system uses a model trained using tool usage data, enables learning and execution based on actual behavior, and is suitable for the automation of repeatable GUI tasks. However, since the LAM system requires a training process, there are limitations to generalization.

[0120] These AI systems are summarized as shown in [Table 3] below.

[0121] [Table 3]

[0122]

[0123] FIG. 20 is a diagram showing an AI agent memory structure according to one embodiment of the present disclosure.

[0124] Referring to FIG. 20, the process of an AI agent generating a response to a query by utilizing memory is illustrated. Memory is divided into short-term and long-term, and each can support complex decision-making and task execution through various types of memory. In one embodiment, memory may consist of short-term memory and long-term memory. Short-term memory is a temporary memory space activated during work, which can focus on the currently ongoing workflow (task execution). Short-term memory may include Working Memory, which manages reasoning and task flow per workflow, and Cache Memory, which provides rapid access to frequently used data and result values. Long-term memory is knowledge and experience-based memory that is continuously preserved, and may include Episodic Memory, which stores events or incidents manually saved in a specific workflow; Semantic Memory, which stores conceptual and factual knowledge (e.g., "Paris is the capital of France"); and Procedural Memory, which stores methods of task execution or procedural knowledge (e.g., "How to reset a server").

[0125] In one embodiment, all memory can operate in conjunction with a Language Model Framework through a central Memory controller. An input query can go through a process via the framework of 1) query analysis and memory referencing, 2) retrieving relevant information from memory if necessary, 3) the language model generating a response through a decision procedure, and 4) delivering the response result to the user.

[0126] In one embodiment, the MCP server is responsible for interfacing with external knowledge and tools and may include a vector database which is a search-based embedding vector repository, a Semantic Database which is a conceptual knowledge base database, a third-party API integration unit, etc.

[0127] In other words, the process can be handled as follows.

[0128]

[0129] One embodiment of the present disclosure can systematically integrate design elements essential for implementing an agent system that can continuously learn and make decisions according to the situation, as shown in [Table 4] below.

[0130] [Table 4]

[0131]

[0132] Next, I would like to explain the types of language models utilized by AI agent systems. Each model has a specific processing method and role, and a suitable model can be selected and utilized depending on the nature of the task.

[0133] FIG. 21 is a drawing showing a GPT (General Pretrained Transformer) model according to one embodiment of the present disclosure.

[0134] Referring to Fig. 21, the GPT model is a general-purpose language model pre-trained on a large text corpus, capable of tokenizing and embedding input prompts, generating hidden states through a transformer layer, calculating logits and probabilities for the next token, and sequentially sampling or selecting tokens to generate text. This GPT model has the advantage of high generality.

[0135] FIG. 22 is a drawing showing a Mixture of Experts (MoE) model according to one embodiment of the present disclosure.

[0136] Referring to Fig. 22, the Mixture of Experts (MoE) model is a decentralized model structure in which only some experts (sub-networks) selected based on the input are activated. The input is tokenized and embedded, a gating network selects a top expert sub-model per token, the outputs of the selected experts are merged (weighted average or aggregated), and decoding is performed based on the merged result. This MoE model has the advantage of high computational efficiency relative to the number of parameters.

[0137] FIG. 23 is a drawing showing a Large Reasoning Model (LRM) according to one embodiment of the present disclosure.

[0138] Referring to FIG. 23, the LRM is a model capable of processing a chain of thought for complex reasoning, performing input and context tokenization, internally generating a reasoning path, evaluating or regenerating possible logical paths, and determining and outputting a final logical answer. Such an LRM may be suitable for solving high-difficulty problems, logic-based question answering, etc.

[0139] FIG. 24 is a drawing showing a Vision Language Model (VLM) according to one embodiment of the present disclosure.

[0140] Referring to FIG. 24, VLM is a multimodal model that integrates and processes images and text. It can perform image encoding, perform text tokenization, combine both modalities into an integrated embedding, and generate a response by inferring based on the integrated representation. VLM can be advantageous for generating explanations and answering questions that include visual information.

[0141] FIG. 25 is a drawing showing a Small Language Model (SLM) according to one embodiment of the present disclosure.

[0142] Referring to FIG. 25, the SLM is a language model with a lightweight structure that can be used in environments with limited computational resources. It performs input tokenization, projects embedded tokens into a low-dimensional space, passes them through a simplified transformer layer, calculates token probabilities, and generates outputs. Such an SLM can be suitable for edge devices, on-device AI environments, etc.

[0143] FIG. 26 is a drawing showing a Large Action Model (LAM) according to one embodiment of the present disclosure.

[0144] Referring to FIG. 26, LAM is a model trained to perform actions in a real environment, capable of tokenizing and embedding task descriptions and environmental states as inputs, planning a sequence of actions (based on Chain-of-Thought), executing actions and making API calls within the environment, and monitoring results. LAM can perform modification and iteration procedures as needed. In one embodiment, LAM can be used in robot control, game agents, automation systems, etc.

[0145] FIG. 27 is a drawing showing a Hierarchical Reasoning Model (HRM) according to one embodiment of the present disclosure.

[0146] Referring to Fig. 27, HRM is a hierarchical model that processes inference by dividing it into high-level planning (H-layer) and low-level computation (L-layer). It can derive a final result by performing high-level planning after input encoding, performing detailed operations and iterations at each step, and constructing an iterative feedback loop until convergence. HRM can be effective for complex multi-step planning and inference tasks.

[0147] FIG. 28 is a drawing showing a ToolFormer (Tools-trained Model) according to one embodiment of the present disclosure.

[0148] Referring to Fig. 28, ToolFormer is a language model trained to use various external tools. It starts based on a pre-trained LLM, samples examples of tool calls, determines valid tool usage through test and evaluation, and performs fine-tuning with filtered data. ToolFormer has advantages in integration with calculators, searches, API calls, etc.

[0149] The four major types of artificial intelligence systems—basic LLM workflow, RAG (Retrieval-Augmented Generation), single AI agent, and multi-agent based Agentic AI—are compared and summarized in terms of structure, function, characteristics, and use cases as shown in [Table 5] and [Table 6] below.

[0150] [Correction pursuant to Rule 91 Nov. 18, 2025]

[0151] [Correction pursuant to Rule 91 Nov. 18, 2025]

[0152] [Table 6]

[0153]

[0154] FIGS. 29 to 34 are drawings illustrating vulnerabilities of an MCP according to an embodiment of the present disclosure. Referring to FIG. 29, a command injection problem may occur in the MCP. That is, a hidden command (intended meaning) can be inserted into a prompt entered by a user to induce an agent to operate on the MCP server without authorization. For example, the agent may gain unauthorized access to external resources such as drant or supabase.

[0155] Referring to Fig. 30, tool addiction problems can occur in MCPs. By including a tool with malicious code inserted into the MCP, it can be induced to produce incorrect results for specific tasks or perform intended actions. For example, an attacker can gain access to a service such as Slack and steal API keys or personal information.

[0156] Referring to Figure 31, server-sent event issues can occur in MCP. Since the Server-Sent Events (SSE) method transmits data in segments, the connection must remain open for a long time, which can cause latency and security issues. For example, when transmitting to Slack, drant, stride, etc., the connection may be maintained for a long time, which can lead to security risks.

[0157] Referring to Fig. 32, privilege escalation issues can occur in MCP. A malicious tool can intercept or overwrite calls to other trusted tools, thereby stealing the privileges of the tools trusted by the user. For example, an attacker can override tools such as Slack within the MCP server to connect to a malicious server.

[0158] Referring to Fig. 33, persistent context issues can occur in MCP. MCP records and maintains context throughout a user's session. This can lead to context tampering. For example, there is a risk that session contexts linked with tools such as AWS, Kagi, and Notion may be tampered with.

[0159] Referring to Fig. 34, a problem of server data theft may occur in MCP. If a tool server connected to the MCP server is hacked, data and passwords from other servers can be stolen or controlled. For example, a client may be able to access user data from another server through a malicious MCP tool.

[0160] FIG. 35 is a diagram illustrating a context engineering structure in an AI agent system according to one embodiment of the present disclosure.

[0161] Referring to Fig. 35, a flow and memory structure can be used to effectively configure and utilize context in an AI Agent system. User input may include input queries or requests provided by the user to the system. The agent may be a central component that plans tasks and coordinates execution based on the given input and context. Additionally, the RAG (Retrieval-Augmented Generation) may include a knowledge retrieval component that retrieves relevant documents through vector search based on long-term memory. Action Tools may perform functions such as calling various external tools for code execution, document lookup, time checking, and calendar processing. Long-Term memory includes memory based on persistent knowledge stored through an MCP server and database, and Short-Term memory may store prompt components containing the context of the current conversation session (input, reasoning, tool usage history, etc.).

[0162] In one embodiment, a user may input a question or request into the system. Based on user input, the agent may formulate a plan and coordinate tasks according to the current context and purpose. The RAG system may perform a search on the vector DB if necessary and collect relevant information or documents. Action Tools may call external tools according to the requested task to retrieve execution results. The prompt generation and update unit may generate or update prompts based on the collected information (search, tools, results, inference, etc.). A final response may be generated and delivered to the user following these procedures. All contextual elements may be stored in the 'Chat History' within short-term memory. For example, information related to input, tools, usage, inference, etc., may be stored in short-term memory. Specific contexts, results, etc., may be added to long-term memory (e.g., MCP server / DB) to ensure reusability.

[0163] In one embodiment, user input includes initial input such as user questions or instructions, and tools may include external systems, APIs, calculators, document tools, etc. Agents may perform internal judgment, planning, state management, etc. as inference. RAG context includes document-based knowledge through vector search, and user children may include user settings, profiles, IDs, etc. Conversation history may include records of previous queries and responses.

[0164] According to one embodiment, context-based accuracy can be improved by utilizing both short-term memory (prompt) and long-term memory (DB). Additionally, a dynamic prompt can be configured by integrating not only user input but also tool usage results, search documents, and reasoning content. Furthermore, memory layers can be separated. For example, short-term memory is maintained during the session, while long-term memory can be used for long-term strategic iterations. An agent is responsible for the decision-making and execution of each step and can play a key role in coordinating the overall flow.

[0165] In one embodiment, an agentic AI system centered on user input, including search (RAG), tool calling, internal reasoning, and memory integration, may be utilized. The system aims to provide sophisticated responses and perform tasks tailored to the situational context by sophisticatedly structuring and utilizing various contextual information generated during interaction with the user. In one embodiment, user input includes various forms of user requests such as text, voice, and images; the agent module performs planning, determines whether to search or execute a tool, and generates responses based on user requests; and the RAG module performs vector-based similar document searches and can be used for external knowledge and context reinforcement. Action tools can call various functional tools such as external APIs, calculators, calendars, search engines, and databases. In one embodiment, the prompt engine can construct a final prompt by integrating various contextual elements. Short-Term Memory (STM) is a temporary storage for maintaining context and constructing prompts within a session, and Long-Term Memory (LTM) is a persistent memory structure based on an MCP server and database, which can be used for the agent's long-term learning and the utilization of accumulated experience.

[0166] The context processing pipeline can be as follows.

[0167]

[0168] In one embodiment, the prompt may be composed of user input corresponding to a query or command, tool usage results including API call results, calculation results, etc., search-based context including documents retrieved from the RAG, agent reasoning including internal reasoning and planning, user information including preferences, ID, status, etc., conversation history including the previous conversation context, etc.

[0169] In one embodiment, the prompt may be deleted based on priority when the maximum prompt length is exceeded. For example, the priority may be configured in the order of agent inference > search context > tool results > user information > past conversation history. In one embodiment, result values ​​after a tool call may be inserted into the prompt in the form of a "contextual tag." Additionally, the search context may be inserted along with a summary and confidence score, rather than the original text.

[0170] In one embodiment, short-term memory is intended to maintain the entire context within a session and may store user input, prompt components, reasoning processes, tool usage results, etc. Short-term memory is deleted upon session termination, but important information may be transferred to long-term memory.

[0171] In one embodiment, the long-term memory may be composed of an MCP server (Agent Metadata) and a domain knowledge DB (Structured Knowledge), etc. The long-term memory may be updated when the Add to memory command is executed or when automatic saving conditions are satisfied. For example, information such as "User A prefers tools related to 'data visualization'" and "On August 7, 2025, the 'RAG + Tools' path was used in the 'Context Engineering' flow" may be stored in the long-term memory.

[0172] In one embodiment, the agent can determine whether to call a tool, the necessity of searching, and the possibility of repeated calls. Additionally, the agent can perform priority-based reasoning. For example, priorities may proceed in the order of user goal → environment state → available resources → execution strategy. In one embodiment, the agent can support parallel calls to multiple tools and support feedback-based iterative execution after execution.

[0173] For example, in the case of multimodal question and answer, when a user image is uploaded, a corresponding description is generated, and a date corresponding to the user image can be calculated via a tool call.

[0174] As another example, in the case of report generation, the search and summarization process proceeds based on user instructions, and templates can be inserted and edited.

[0175] As another example, in the case of automated schedule coordination, the calendar API is invoked based on natural language requests, and schedule recommendations can be provided.

[0176] According to one embodiment, a multi-agent-based Agentic AI extension structure may be supported. Additionally, according to one embodiment, a prompt dynamic optimization (auto-slimming) algorithm may be implemented. Furthermore, a memory vectorization-based summary storage module may be constructed, and user-specific customized context weighting profiling may be performed.

[0177] In particular, AI agents are characterized by autonomy and independence. Autonomous agents act according to their own rules without external intervention; a representative example is Automated Guided Vehicles (AGVs) in the robotics field, which recalculate their own paths based on real-time order data. Furthermore, they possess decentralized characteristics, allowing agents to make independent decisions without central control. This is implemented in the robotics field by multi-robot systems establishing collaborative path planning using only local sensor data.

[0178] One of the characteristics of AI agents is that they act from a local perspective. Local views is a key concept in AI agent technology, referring to the characteristic where an agent perceives only the parts related to its mission, rather than the entire system. This is a core element that enables efficient decision-making and resource utilization in distributed systems. Agents perceive only the parts relevant to their tasks, and in cyber-physical systems (CPS), this is implemented in a way where each agent monitors only a specific section of the power grid. This local perspective enables efficient decision-making and resource utilization in distributed systems.

[0179] 1. Limitation of Agent's Scope of Awareness

[0180] The local perspective works effectively in large-scale, complex systems where it is impossible for agents to grasp the entire system state simultaneously. Each agent processes information by focusing only on the specific area or task responsible for them.

[0181] For example, in a cyber-physical system (CPS), each agent in the power grid monitors only a specific section. As another example, in a multi-drone system, each drone focuses on its own path and surrounding environment.

[0182] 2. Advantages of Distributed Systems

[0183] The local perspective offers the following benefits in distributed systems: increased system scalability and flexibility; reduced cognitive load on individual agents; improved resource efficiency; and the ability to make effective decisions in complex systems.

[0184] 3. Implementation Examples in AI Systems

[0185] Regional perspectives are being implemented in various AI systems:

[0186] - Robotics Field: Multi-robot systems establish collaborative path planning using only local sensor data

[0187] - Smart Grid: Distributed agents jointly perform power demand forecasting and energy distribution optimization

[0188] - Autonomous driving: In distributed reinforcement learning-based systems such as DriveTransformer, each sensor collects local data

[0189] A regional perspective can increase the efficiency of large-scale systems in the following ways.

[0190] - Saves computational resources because it is not necessary to monitor the entire system status simultaneously.

[0191] - Optimize only the parts handled by individual agents

[0192] - Achieve overall system optimization through collaboration between agents

[0193] The local perspective is a core principle of distributed systems and can enhance the following characteristics:

[0194] - Decentralization: Agents perform decisions independently without central control

[0195] - Autonomy: Acting according to its own rules without external intervention

[0196] - Scalability: Even as the system size increases, the burden on each agent is relatively low.

[0197] A major challenge that arises when agents with a local perspective cooperate in distributed systems is the complexity of designing reward structures to achieve the overall system goal while each agent learns independently; in recent research, approaches utilizing game-theoretic methods to design cooperation mechanisms between agents are gaining attention.

[0198] Additional challenges arising from the limitations of the regional perspective include the need for strategies for information sharing and consistency maintenance among agents, as well as the necessity of research on effective utilization methods for the extensive information required for the optimization of the entire system.

[0199] The regional perspective is evolving as follows with the advancement of multimodal AI agents:

[0200] - AI agents are evolving into multimodal AI agents that integrally process various modal data such as images, text, and audio.

[0201] - Particularly in the field of autonomous driving, development is moving toward multimodal AI agents based on Vision Transformers processing driving environments end-to-end.

[0202] Along with the implementation of a regional perspective, the following research is being conducted:

[0203] The development of explainable AI agents, capable of explaining the decision-making processes of AI agents, is emerging as an important research task.

[0204] This is expected to contribute to enhancing the reliability and transparency of AI agents.

[0205] Local Views is a characteristic in which an AI agent perceives only the parts related to its mission rather than the entire system, enabling efficient decision-making and resource utilization in distributed systems. This concept is being applied in various fields such as robotics, autonomous driving, and smart grids, and is a key element in enhancing the scalability and flexibility of distributed systems. Recent research is evolving toward the development of multimodal AI agents and explainable AI agents based on this concept. These technical characteristics and evolution are advancing in a direction that further enhances the autonomy, decentralization, and efficiency of AI agents.

[0206] Recently, Decentralized Reinforcement Learning (DRE) has established itself as the standard architecture for autonomous systems in the field of AI agents. DRE is characterized by each agent independently perceiving its environment and making distributed decisions based on local data. This approach is being applied in various fields, including autonomous driving, multi-robot systems, and smart grids.

[0207] In the field of autonomous driving, the 'DriveTransformer' framework is a distributed reinforcement learning-based path planning system that utilizes a Transformer architecture to process driving environments end-to-end. DriveTransformer integrates and processes data from various sensors, such as cameras, LiDAR, and radar, and can autonomously calculate the optimal driving path by considering road conditions and traffic regulations.

[0208] A major challenge in distributed reinforcement learning is the complexity of designing cooperative rewards. Designing an appropriate reward structure is crucial for each agent to learn independently while achieving the overall system's goal. To this end, approaches that utilize game-theoretic methods to design cooperation mechanisms between agents are gaining attention.

[0209] Multi-Agent Systems are an extended form of AI agent technology, referring to systems where multiple agents interact to achieve a common goal. Multi-Agent Systems possess the following three main characteristics.

[0210] First, decentralization. In multi-agent systems, agents make decisions independently without central control. For example, in the field of robotics, multi-robot systems establish collaborative path planning using only local sensor data. This decentralization has the advantage of increasing the scalability and flexibility of the system while reducing the cognitive load of individual agents.

[0211] Second, local views. In a multi-agent system, each agent perceives only the part related to its mission, rather than the entire system. In cyber-physical systems (CPS), this is implemented by having each agent monitor only a specific section of the power grid. This local view enables effective operation in large-scale, complex systems where it is impossible to simultaneously grasp the state of the entire system.

[0212] Third, autonomy. Agents in a multi-agent system act according to their own rules without external intervention. For example, Automated Guided Vehicles (AGVs) in a warehouse automation system recalculate their own routes based on real-time order data. This autonomy is a key factor in enhancing the system's adaptability and efficiency.

[0213] Smart grids are an application example of multi-agent systems. In smart grids, distributed agents jointly perform power demand forecasting and energy distribution optimization, operating via trust-based collaboration protocols. Additionally, in the field of autonomous robot collaboration, multiple drones can improve operational efficiency by simultaneously calculating obstacle avoidance paths using distributed reinforcement learning.

[0214] Designing a neural network architecture for AI agents requires a specialized structure that considers the agent's characteristics and operating environment. Since AI agents follow a structure of Perception → Knowledge Base → Reasoning & Planning → Action & Actuation → Learning Loop, designing a neural network architecture that supports this is crucial.

[0215] The DriveTransformer study is a prime example that effectively reflects the characteristics of such agent architectures. Based on the Transformer architecture, DriveTransformer supports end-to-end processing of autonomous driving agents and enables autonomous decision-making and learning when combined with a distributed reinforcement learning framework.

[0216] DriveTransformer is an autonomous driving AI model and a distributed reinforcement learning-based path planning system that utilizes a Transformer architecture to process driving environments in an end-to-end manner. This system integrates and processes data from various sensors, such as cameras, LiDAR, and radar, and can autonomously calculate the optimal driving path by considering road conditions and traffic regulations.

[0217] DriveTransformer is designed based on Decentralized Reinforcement Learning and has the following features:

[0218] - Each agent independently perceives the environment and performs distributed decision-making based on local data.

[0219] - It can be applied to various fields such as autonomous driving, multi-robot systems, and smart grids.

[0220] - Cooperative reward design is a major challenge, and the design of cooperative mechanisms utilizing game theory methods is applied.

[0221] An example of applying the Vision Transformer (ViT) to an autonomous driving environment, reflecting the direction of the Vision Transformer's evolution:

[0222] - ViT-based models have the disadvantage of high computational costs, so a lightweight and efficient architecture design is applied.

[0223] - The LLM pre-training method was innovated by introducing a multi-token prediction technique, which can also contribute to improving the efficiency of ViT.

[0224] DriveTransformer reflects the characteristics of a multi-agent system, namely Autonomy, Decentralization, and Local Views.

[0225] DriveTransformer follows the basic structure of an AI agent: 'Perception → Knowledge Base → Reasoning & Planning → Action & Actuation → Learning Loop'.

[0226] 1. Perception

[0227] It recognizes the surrounding environment by integrally processing data from various sensors, such as cameras, lidar, and radar.

[0228] 2. Memory (Knowledge Base)

[0229] It stores recognized environmental information and plays a role in familiarizing itself with traffic conditions, road structures, traffic regulations, etc.

[0230] 3. Reasoning (Reasoning & Planning)

[0231] It utilizes a Transformer-based model to calculate the optimal driving path and make situational decisions.

[0232] 4. Execution (Action & Actuation)

[0233] Controls the vehicle and executes driving according to the determined path.

[0234] 5. Learning Loop

[0235] It receives feedback on driving results to continuously learn and improve performance.

[0236] According to one embodiment of the present disclosure, a driving path can be planned directly without complex intermediate steps through end-to-end processing.

[0237] According to one embodiment of the present disclosure, various sensor data are processed integrally to enable more accurate driving decision-making.

[0238] According to one embodiment of the present disclosure, multiple agents can cooperate through distributed reinforcement learning to build a more efficient driving environment.

[0239] One embodiment of the present disclosure can be applied to the field of autonomous driving. It integrates and processes various sensor data and can autonomously calculate an optimal driving path considering road conditions and traffic regulations.

[0240] One embodiment of the present disclosure can be applied to the field of robotics. In various industrial fields such as logistics centers, construction sites, and agriculture, a multi-robot system can establish a collaborative path plan using only local sensor data.

[0241] One embodiment of the present disclosure may be applied to the field of energy management. It can be applied to various energy management systems, such as power demand forecasting and energy distribution optimization.

[0242] DriveTransformer is expected to evolve in the following directions based on current technology:

[0243] - Advancement of inter-agent cooperation protocols: As the design of cooperation mechanisms utilizing game-theoretic methods further develops, effective cooperation in complex multi-agent systems will become possible.

[0244] - Evolution into Multimodal AI Agents: AI agents will evolve into multimodal AI agents that integrally process various modal data, such as images, text, and audio.

[0245] - Development of Explainable AI Agents: The development of explainable AI agents, capable of explaining the decision-making process of AI agents, is emerging as an important research task.

[0246] DriveTransformer is expected to evolve into a more efficient and safer autonomous driving system by following this direction of development.

[0247] The key considerations for designing neural network architectures for AI agents are as follows: First, efficient learning in distributed environments. Since distributed agent systems learn independently in different environments, an architecture design capable of effective learning even in distributed settings is required. Second, the integration of environment perception and decision-making. Because agents must perceive their environment and make decisions based on this, a structure capable of processing environment perception and decision-making in an integrated manner is crucial. Third, a structure that supports collaborative learning. In multi-agent systems, cooperation among agents is critical, so an architecture design that supports collaborative learning is necessary.

[0248] Improving the efficiency of neural network architectures is crucial, particularly when implementing AI agents in mobile and embedded environments. An LLM pre-training method can be provided by introducing multi-token prediction techniques. This approach can significantly improve computational efficiency by predicting multiple tokens simultaneously instead of relying on traditional token-based prediction.

[0249] Research on neural network architectures for efficiency improvement is proceeding in three main directions. First, regarding model lightweighting, lightweight model architectures are being developed that reduce the number of parameters and computational load, enabling effective learning and inference with fewer resources.

[0250] Second, research is underway to improve the efficiency of the entire system through efficient distributed learning and inference by efficiently distributing data and computations across multiple devices.

[0251] Third, regarding on-device learning and inference, the design of neural network architectures for implementing on-device AI agents, in which models collect data in real-time from actual environments and perform learning and inference based on this data, is actively underway.

[0252] One embodiment of the present invention relates to artificial intelligence (AI) agent technology, and in particular, to a technology that implements an efficient decision-making and cooperation mechanism utilizing the characteristics of a decentralized reinforcement learning-based autonomous system architecture and a multi-agent system. An AI agent is a computer system implementation of human decision-making and behavioral processes, and its basic structure follows a cyclical structure of 'Perception → Knowledge Base → Reasoning & Planning → Action & Actuation → Learning Loop'. Through this structure, the AI ​​agent perceives the environment, makes decisions based on collected information, executes the results, and learns from those results to continuously improve.

[0253] One embodiment of the present invention is a technology for implementing an AI agent with enhanced autonomy, centered on a distributed reinforcement learning architecture that independently perceives the environment without central control and performs decision-making based on local data. This can be applied to various fields such as autonomous driving, multi-robot systems, and smart grids, and specifically enables efficient decision-making and resource utilization in large-scale, complex systems by leveraging the characteristic that each agent perceives only the parts related to its mission from a local perspective.

[0254] Decentralized Reinforcement Learning (Decentralized RL) has recently established itself as the standard architecture for autonomous systems in the field of AI agents, characterized by each agent independently perceiving the environment and making distributed decisions based on local data. This approach is being applied in various fields such as autonomous driving, multi-robot systems, and smart grids, and in particular, enables real-time communication and collaboration between multiple platforms through innovative designs such as the MAES framework.

[0255] Decentralized Reinforcement Learning is a learning methodology in which multiple agents independently perceive the environment and make distributed decisions based on local data. Unlike traditional centralized reinforcement learning, it is characterized by each agent processing and learning data individually.

[0256] Distributed reinforcement learning has the following key characteristics:

[0257] - Decentralization: Each agent learns and makes decisions independently without a central server or controller.

[0258] - Local Views: Each agent perceives only information related to their own mission, rather than the entire system.

[0259] - Autonomy: Acting according to its own rules without external intervention

[0260] Distributed reinforcement learning is being applied in various fields, including autonomous driving, multi-robot systems, and smart grids. Particularly in the field of autonomous driving, the 'DriveTransformer' framework is a distributed reinforcement learning-based path planning system that utilizes a Transformer architecture to process driving environments end-to-end. DriveTransformer integrates and processes various sensor data, such as cameras, LiDAR, and radar, and can autonomously calculate the optimal driving path by considering road conditions and traffic regulations.

[0261] A major challenge in distributed reinforcement learning is the complexity of cooperative reward design. It is important to design an appropriate reward structure so that each agent learns independently while achieving the goal of the overall system. To this end, in one embodiment, a cooperation mechanism between agents can be designed by utilizing game-theoretic methods.

[0262] One of the directions for the development of distributed reinforcement learning is the advancement of cooperation protocols between agents. As the design of cooperation mechanisms utilizing game-theoretic methods further develops, effective cooperation in complex multi-agent systems can become possible.

[0263] Distributed reinforcement learning is advancing through the evolution into multimodal AI agents. AI agents can evolve into multimodal agents that integrally process various modal data, such as images, text, and audio. Particularly in the field of autonomous driving, multimodal AI agents based on Vision Transformers are evolving to handle driving environments end-to-end.

[0264] Distributed reinforcement learning systems are expected to advance further by combining with high-performance hardware. In particular, research on combining them with hardware accelerators to efficiently perform large-scale distributed learning is actively underway.

[0265] In the field of autonomous driving, distributed reinforcement learning is being used to implement systems where each vehicle or autonomous driving system independently perceives its surroundings and determines optimal driving strategies while contributing to the overall traffic flow. DriveTransformer is an example of this distributed reinforcement learning, capable of autonomously calculating the optimal driving path by integrally processing various sensor data.

[0266] In robotic systems, multiple robots establish a collaborative path plan using only local sensor data. For example, in a logistics center, Automated Guided Vehicles (AGVs) recalculate their own paths based on real-time order data to support efficient logistics processing. Additionally, multiple drones can improve operational efficiency by simultaneously calculating obstacle avoidance paths using distributed reinforcement learning.

[0267] In the smart grid sector, distributed agents jointly perform power demand forecasting and energy distribution optimization, operating via trust-based collaboration protocols. Each agent monitors a specific section of the power grid and, based on this, can optimize the entire system.

[0268] Distributed reinforcement learning is a learning methodology in which each agent learns and makes decisions independently without centralized control, and it is being applied in various fields such as autonomous driving, robotics, and smart grids. A key feature of distributed reinforcement learning is enhancing system scalability and flexibility through a decentralized architecture. A major challenge is designing cooperation mechanisms between agents, and game-theoretic methods are being utilized for this purpose. In the future, it is expected to further advance through the sophistication of cooperation protocols between agents, improved multimodal data processing capabilities, and integration with high-performance hardware.

[0269] Several major issues exist in AI agent systems, particularly in autonomous systems based on distributed reinforcement learning. First, the complexity of designing cooperative rewards between agents presents a serious problem. Designing an appropriate reward structure is essential for each agent to learn independently while achieving the overall system's goal; however, the complexity arising from this process acts as a major barrier to system design and implementation. To address this, the design of cooperative mechanisms utilizing game-theoretic methods has been proposed, but issues regarding complexity and high computational costs still persist.

[0270] Second, in large-scale multi-agent systems, the impact of inter-agent communication and cooperation mechanisms on performance is significant. Particularly in fields where real-time processing is critical, such as autonomous driving and robotics, communication delays and overhead can impede the overall system performance. While the decentralization characteristic, where agents make decisions independently without central control, offers the advantages of increasing system scalability and flexibility while reducing the cognitive load of individual agents, it also presents the problem of difficulty in achieving an optimal state for the entire system due to information asymmetry and imbalance.

[0271] Third, in multi-agent systems characterized by local views, there is a problem in achieving global optimization because each agent cannot fully grasp the state of the entire system. In cyber-physical systems (CPS), each agent is implemented by monitoring only specific sections of the power grid; however, the resulting information gaps can impede the efficiency and stability of the entire system.

[0272] Fourth, while autonomous AI agents have the characteristic of acting according to their own rules without external intervention, this autonomy can lead to unpredictability and safety issues. In particular, for autonomous driving systems such as Automated Guided Vehicles (AGVs) that recalculate routes based on real-time order data, safety margins and exception handling become critical factors when designing the system.

[0273] One embodiment of the present invention can present various approaches to solve the aforementioned problems. First, the problem of designing complex reward structures can be solved by utilizing game-theoretic methods when designing a cooperation mechanism between agents. In particular, by designing a cooperation mechanism so that each agent can learn independently while achieving the goal of the entire system in a decentralized reinforcement learning (DRL) environment, the problem of reward conflict between agents can be minimized.

[0274] Second, one embodiment of the present invention may apply distributed learning techniques, such as Federated Learning, to improve the communication efficiency of a multi-agent system. Federated Learning operates by having each agent train a model based on local data and then transmitting the learned parameters to a central server for integration; in this process, efficient information exchange and cooperation between agents can be enabled. In particular, A2A (Agent-to-Agent Protocol) is a standard communication protocol between agents developed by Google, which can be used for autonomous AI agents to communicate in a common language, exchange information, and coordinate cooperative tasks.

[0275] Third, one embodiment of the present invention can develop a cooperation algorithm that utilizes the local views characteristics of a multi-agent system. A hierarchical cooperation structure and an information sharing protocol can be designed so that each agent can contribute to the overall system's goal while performing actions optimized within its own local environment. Through this, efficient cooperation and optimization can be achieved even in a distributed system without central control.

[0276] Fourth, one embodiment of the present invention can implement a hybrid control system to simultaneously ensure autonomy and safety. For example, an Automated Guided Vehicle (AGV) of a warehouse automation system can have a redundant decision-making structure that recalculates its own route based on real-time order data, while also enabling safety control by linking with a central monitoring system in high-risk situations.

[0277] Fifth, one embodiment of the present invention may introduce a mechanism capable of interpreting the decision-making process of a system through the development of an Explainable AI Agent. This contributes to enhancing the transparency and reliability of autonomous systems and can be a significant factor, particularly in strictly regulated fields such as healthcare and finance.

[0278] One embodiment of the present invention includes the following innovative elements. First, a cooperative and efficient multi-agent system can be implemented without central control through a decentralized reinforcement learning (Decentralized RL)-based autonomous system architecture. This offers superior scalability and flexibility compared to existing centralized control systems and operates effectively, particularly in large-scale complex systems.

[0279] Second, by implementing a multimodal AI agent based on a Transformer architecture such as DriveTransformer, it is possible to integrally process various sensor data (cameras, LiDAR, radar, etc.) and autonomously calculate the optimal driving path considering road conditions and traffic regulations. This can be an innovative method for processing the driving environment in an end-to-end manner in the field of autonomous driving.

[0280] Third, by providing a standardized protocol for real-time communication and collaboration between multiple platforms, such as the MAES framework, multi-robot systems in the robotics field can establish collaborative path planning using only local sensor data. This can be a significant achievement applicable to various industrial sectors, such as logistics centers, construction sites, and agriculture.

[0281] Fourth, as a practical application, multiple drones can improve work efficiency by simultaneously calculating obstacle avoidance paths using distributed reinforcement learning. This has the potential to be applied to various fields such as agricultural pest control, disaster monitoring, and 3D terrain modeling.

[0282] Fifth, a game-theoretic approach to the advancement of inter-agent cooperation protocols can make an innovative contribution to the design of efficient cooperation protocols in complex multi-agent systems. This can resolve the complexity issues of existing distributed reinforcement learning.

[0283] The future direction of development for the present invention is as follows. First, it should develop in a direction that further strengthens the balance between autonomy and safety. In particular, in fields where safety is critical, such as autonomous driving and robotics, it is necessary to further refine the handling of exceptional situations and emergency response mechanisms.

[0284] Second, distributed collaboration mechanisms must be further developed to enhance the scalability of multi-agent systems. In particular, scaling methodologies are needed that can operate efficiently even in large-scale systems where the number of agents increases exponentially.

[0285] Third, lightweight protocols and communication mechanisms must be developed to maximize communication efficiency between agents. In particular, technology that enables real-time collaboration while minimizing communication costs in wireless network environments is important.

[0286] Fourth, we must further develop Explainable AI Agent technology to enhance the transparency and reliability of decision-making processes. This is a particularly important factor in strictly regulated fields such as healthcare and finance.

[0287] Fifth, we must advance Multi-modal AI Agent technology to build a system capable of integrally processing not only various sensor data but also diverse forms of data such as text, images, and voice. This will enable innovative applications, particularly in various fields such as healthcare, entertainment, and education.

[0288] The fields of autonomous driving and robotics are the most representative application areas for AI agent technology. As previously mentioned, the DriveTransformer research demonstrates the advancement of autonomous driving systems utilizing the Transformer architecture. This system integrates and processes data from various sensors, such as cameras, LiDAR, and radar, and autonomously calculates the optimal driving path considering road conditions and traffic regulations.

[0289] In the field of robotics, multi-robot systems are implemented by establishing collaborative path planning using only local sensor data. Such multi-robot systems can be utilized in various industrial sectors, including logistics centers, construction sites, and agriculture. For example, in logistics centers, Automated Guided Vehicles (AGVs) support efficient logistics processing by autonomously recalculating routes based on real-time order data.

[0290] Multiple drones can improve work efficiency by simultaneously calculating obstacle avoidance paths using distributed reinforcement learning. Such multi-drone systems can be applied to various fields, including agricultural pest control, disaster monitoring, and 3D terrain modeling.

[0291] The smart grid sector is a significant application area for multi-agent systems. In smart grids, distributed agents jointly perform power demand forecasting and energy distribution optimization, operating via trust-based collaboration protocols. Each agent monitors a specific section of the power grid and establishes an optimal power distribution plan based on this data.

[0292] Cooperation among agents in a smart grid plays a crucial role in maintaining the balance of power supply and demand and optimizing energy efficiency. For example, if power demand in a specific area surges, agents in adjacent areas can cooperate to efficiently distribute power and minimize energy waste.

[0293] Furthermore, multi-agent systems play a crucial role in distributed energy systems utilizing renewable energy (solar, wind, etc.). Each agent monitors the output of the renewable energy facilities under its management and determines how they are connected to the power grid based on this data. Such systems contribute to effectively managing the variability of renewable energy and maintaining the stability of the power grid.

[0294] The medical and life sciences fields are important application areas for transfer learning. Adaptive technologies for the medical imaging domain and ethical AI application cases are being researched. In the medical field, transfer learning plays a particularly important role due to the scarcity and specificity of patient data.

[0295] Transfer learning techniques can be provided to bridge the real-virtual domain gap by utilizing synthetic vision datasets based on generative models (GAN / Diffusion). This technique helps to effectively train models even when real medical image data is scarce, and improves the accuracy of medical image analysis.

[0296] In particular, AI agent technology is being actively applied in fields such as medical image analysis, disease prediction, and genetic analysis. For instance, medical image analysis agents can analyze patients' CT, MRI, and X-ray images to diagnose diseases and propose treatment plans. These AI agents play a role in providing personalized medical services and supporting the decision-making of medical professionals.

[0297] The fields of finance and business intelligence are important application areas for generative models. Code generation models, particularly RL-Enhanced Code Generation (RLEF), can be applied to various fields such as financial algorithm development, risk management, and the formulation of automated investment strategies. RLEF presents an innovative approach that solves the iterative improvement problem of LLM by combining execution feedback and reinforcement learning, which is of great help in automatically generating and improving complex algorithms in the financial sector.

[0298] Furthermore, generative models are being applied to various fields of financial analysis, such as market forecasting, customer behavior analysis, and fraud detection. For example, image generation models can provide investors with intuitive insights through the visualization of market data.

[0299] In the fields of finance and business intelligence, AI agents play a role in collecting and analyzing market data to propose investment strategies or suggest risk management measures. These AI agents can support the decision-making of financial experts and improve the accuracy and efficiency of business intelligence.

[0300] AI agent technology is evolving into autonomous system architectures based on Decentralized Reinforcement Learning (Decentralized RL). The major future directions for the development of AI agent technology are as follows.

[0301] First, the advancement of cooperation protocols between agents. As the design of cooperation mechanisms utilizing game-theoretic methods is further developed, effective cooperation in complex multi-agent systems will become possible. Decentralized RL, Multi-Robot Systems, and game-theoretic methods may be provided.

[0302] Second, the evolution into multimodal AI agents. AI agents can evolve into multimodal AI agents that integrally process various modal data, such as images, text, and audio. Particularly in the field of autonomous driving, multimodal AI agents based on Vision Transformers can evolve to process the driving environment end-to-end.

[0303] Third, the development of explainable AI agents. The development of explainable AI agents, capable of explaining the decision-making process of AI agents, is emerging as an important research task. This will contribute to enhancing the reliability and transparency of AI agents.

[0304] While generative models have demonstrated innovative achievements in various fields such as image generation and code generation, limitations like spectral bias still exist. Accordingly, the future direction of generative model development is as follows.

[0305] First, the resolution of the spectral mismatch problem. It is important to improve the high-frequency component generation capabilities of Diffusion models and GANs to generate images that are more similar to real images. These limitations can be identified through the analysis of the spectral bias of generative models.

[0306] Second, the improvement of generalization ability across domains. Adversarial training and discrepancy-based methods for generalization between real and virtual synthetic domains can be further advanced to develop generative models that can be effectively applied across various domains.

[0307] Third, improving the efficiency of generative models. As demonstrated by Visual Autoregressive Modeling (VAR), structural innovations that maximize computational efficiency can be sustained. In particular, the development of lightweight generative models capable of efficient inference in distributed environments is an important research task.

[0308] Improving the efficiency and generality of neural network architectures is an important research direction in modern AI technology. Multi-token prediction techniques and SyntaGen research can enhance the efficiency and generality of neural network architectures.

[0309] The future direction of neural network architecture development is as follows.

[0310] First, a lightweight architecture for on-device AI agents. It is important to develop a lightweight neural network architecture that can operate effectively even in mobile and embedded environments.

[0311] Second, an integrated architecture that supports multimodal learning. It is necessary to develop an integrated neural network architecture capable of effectively processing data of various modalities, such as images, text, and audio.

[0312] Third, efficient learning methods through self-supervised learning and meta-learning. There is a need to provide a method for integrating self-supervised learning and meta-learning techniques, which enable effective learning even with limited data, into neural network architectures.

[0313] In one embodiment of the present invention, a distributed reinforcement learning-based AI agent system that autonomously determines a path and performs cooperative actions based on multimodality sensor data input from an environment is provided, wherein the system comprises a plurality of agents, and each agent includes: a sensing unit that collects sensor data within the agent’s assigned area and generates local state information; a memory unit that stores and manages the local state information; an inference unit that performs decision-making and path planning for achieving a goal within the environment based on the local state information; an execution unit that reflects the determined path or action to an external environment; and a learning loop that receives a reward value from the execution result and updates a policy, wherein each agent performs learning and decision-making independently without central control, and the plurality of agents collaborate to achieve a global goal through a game theory-based cooperative reward structure, and the inference unit of the agent includes a Transformer-based neural network architecture to integrally process multimodality data and infer optimal actions for each situation.

[0314] In one embodiment of the present invention, each agent is implemented as at least one of a power grid, an autonomous vehicle, a logistics robot, or a drone, and each agent can reduce computational costs and improve the scalability of the entire system by processing only sensor information in its assigned area without accessing information outside its mission scope.

[0315] In one embodiment of the present invention, the game theory-based cooperative reward structure models the interaction between agents such that the behavior of each agent contributes to the overall system reward, and may include a policy learning algorithm that considers Nash Equilibrium or Pareto optimization.

[0316] In one embodiment of the present invention, the learning loop of each agent is configured in an on-device learning or federated learning manner, and the entire model can be updated without external communication or optionally based only on communicated parameters.

[0317] In one embodiment of the present invention, the reasoning unit of the agent may be configured to include an Explainable AI module so as to be able to output the cause of each decision result externally.

[0318] In one embodiment of the present invention, each agent includes a Transformer-based cognitive module and can infer a high-dimensional representation of each environmental state through multi-token prediction or tokenization of visual input and reflect it in action decisions.

[0319] A distributed reinforcement learning-based AI agent system according to an embodiment of the present invention, which collects multimodality sensor data to perform autonomous path determination and cooperative action execution, comprises: a plurality of AI agents each including a sensing unit that detects various environmental information (image, text, audio, temperature, humidity, pressure, etc.) from sensors (e.g., camera, LiDAR, ultrasonic sensor, temperature sensor, microphone, etc.) installed in their respective designated areas and generates local state information based on said information; each agent includes a memory module (e.g., RAM, SSD, NVMe, etc.) that stores and manages the generated local state information; each agent includes an inference module (e.g., reinforcement learning algorithms such as Transformer-based neural networks, DQN, PPO, etc.) that receives the stored local state information as input, determines actions to achieve environmental goals (e.g., movement to a destination, maximization of energy efficiency, avoidance of risk factors, etc.) within the agent's assigned area, and plans a path; and an execution module that reflects the determined actions in the external environment, such as the movement of a robot arm, the movement path of a drone, the engine control of an autonomous vehicle, or the transport of goods by a logistics robot, and receives a reward value from the external environment according to the execution result (e.g., upon arrival at a destination). It includes a learning loop (e.g., interaction with environment → state update → action selection → reward feedback → policy update) that updates its decision-making policy by receiving input (rewards, rewards for energy saving, rewards for successful risk avoidance, etc.), all of which are performed independently by each agent without a central control unit, and each agent cooperates with others to achieve the global goal of the entire system according to a game theory-based cooperative reward structure (e.g., a reward function designed so that each agent's actions contribute to the total reward of the entire agent system), and the agent's inference module includes a Transformer-based neural network architecture to integrally process multimodal data (e.g., images, text, audio, temperature, humidity, pressure, etc.).A distributed reinforcement learning-based AI agent system characterized by including a function to derive optimal actions for each situation may be provided.

[0320] In one embodiment of the present invention, the agent is implemented as at least one of a power grid, an autonomous vehicle, a drone, and a logistics robot, and each agent can reduce the computational cost of the entire system and improve the scalability of the system by processing only sensor data within its jurisdiction without accessing information outside its jurisdiction (such as data from other agents).

[0321] In one embodiment of the present invention, the cooperative reward structure models the interaction between agents so that the behavior of each agent contributes to the reward of the entire system, and may include a policy learning algorithm that approaches a Nash Equilibrium state (e.g., Multi-Agent Deep Deterministic Policy Gradient, MADDPG) or a policy learning algorithm that considers Pareto optimization.

[0322] In one embodiment of the present invention, the learning loop of an agent is configured as an on-device learning method (collecting, processing, and learning data from a local device) or a federated learning method (collecting data collected from multiple devices for learning, but the actual learning model is distributed across each device), and the entire model may be updated without external communication or based only on optionally communicated parameters (neural network weights, policies, etc.).

[0323] In one embodiment of the present invention, the agent may be configured to include an Explainable AI module in the reasoning process to output the result of an action decision and the cause of the decision (e.g., "The reason this path was chosen is because an obstacle was detected," "The reason this power distribution method was chosen is because efficiency and stability were considered") to the outside.

[0324] In one embodiment of the present invention, the agent's Transformer-based inference module can perform multi-token prediction or perform tokenization on visual input data to infer a high-dimensional environmental state representation (e.g., regional energy usage expressed as [[0.1, 0.2, 0.7], [0.3, 0.4, 0.3]]) and reflect this in action decisions.

[0325] A sensing unit according to one embodiment of the present invention includes a camera, LiDAR, ultrasonic, temperature sensor, etc., an inference module includes a reinforcement learning algorithm such as a Transformer-based neural network or DQN, and an execution module may include a robot arm, movement of a drone, engine control of a vehicle, etc.

[0326] A reward value according to one embodiment of the present invention includes, for example, arrival at a destination, successful risk avoidance, energy saving, etc., and a game theory-based cooperative reward structure includes, for example, a reward function designed such that the actions of each agent contribute to the total reward of the entire agent system, and a policy learning algorithm that approaches a Nash Equilibrium state may include a Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm.

[0327] Federated Learning-based Parameter Aggregator and A2A Protocol

[0328] The system may include a Federated Parameter Aggregator to enable updating the policy of the entire system by aggregating parameters individually learned from multiple agent nodes. Each agent trains a local model using local data, and the aggregator module provides parameter update information in an encrypted form ( After receiving ), it integrates them to form a global policy. In addition, to standardize interactions between agents, an Agent-to-Agent (A2A) communication protocol may be applied, which enables the exchange of messages such as model information sharing, collaborative learning, and resource requests and exchanges according to defined specifications.

[0329] Game Theory-based Cooperative Reward Scheduler

[0330] To facilitate autonomous cooperation among agents, the present invention may introduce a game-theoretic reward mechanism. Specifically, based on the results of actions autonomously chosen by multiple agents, an individual utility function u i Calculate and the optimal strategy for maximizing collective utility (e.g., Nash equilibrium policy) It includes a cooperative reward scheduler that schedules ). This structure enables multi-agent optimization that considers overall performance rather than a single selfish agent.

[0331] Explainable AI (XAI) Module

[0332] According to some embodiments, the present invention may further include an Explainable AI (XAI) module to clearly present the basis for an agent's judgment to a user or an external system. The XAI module analyzes the output and internal state of the inference module to generate explanatory information, such as the reason for a specific action choice, considered characteristics, and past correlation cases, and can transmit this information in the form of an API to an external user interface or an audit system (HMI, Audit Logger, etc.).

[0333] Multimodal Cognitive-Synthesizing Transformer

[0334] The present invention may include a multimodal transformer structure capable of integrated processing of various types of data, such as text, images, and voice. Specifically, the Fusion-Transformer transforms input data of different modalities into different embedding spaces and then integrates them to generate a context representation vector z. This structure can be utilized in various AI applications, such as complex situational awareness, multi-sensor fusion, multimodal question answering, and image captioning.

[0335] Safety monitoring and redundancy control module

[0336] The present invention may additionally include a Safety Supervisor module so that safety can be given top priority even in autonomous systems. The module monitors system status information, risk indicators (r) of the external environment, prediction failure rates, etc., and if a preset threshold is exceeded, it may automatically switch to manual control mode or perform a temporary transfer of control in conjunction with a central control unit. Through this, the reliability and safety of the system can be ensured even in extreme situations where learning-based control may become unstable.

[0337] Lightweight communication protocol

[0338] According to one embodiment, the present invention may utilize a lightweight communication protocol to minimize communication delay and power consumption. For example, the Tx / Rx module may use Edge-Lite MsgPack or a similar lightweight messaging format that limits the message size to 64 bytes or less and maintains an average round-trip latency (RTT) of 5 ms or less. This structure can be effectively applied in edge environments, IoT sensor networks, drone control systems, etc.

[0339] On-device / Federated Learning Dual Learning Mode

[0340] To respond to various operating environments, the present invention can support a dual learning mode capable of switching between an on-device learning mode and a federated learning mode. When local resources are sufficient, the system improves policies using only on-device learning; conversely, when collaboration among multiple devices is required or data diversity is needed, it switches to federated learning mode to periodically integrate the entire policy.

[0341] Self-supervised learning and meta-learning-based boosting engine

[0342] According to one embodiment, the present invention may include a boosting engine based on self-supervised learning and meta-learning to maintain performance in situations where initial training data is limited or where rapid adaptation to a new environment is required. For example, by learning the embedding space structure without external labels through a self-supervised objective function L and improving the speed of adaptation to a new task through a meta-learning algorithm, similar accuracy can be maintained even if data usage is reduced by 30%.

[0343] Functions related to artificial intelligence according to the present disclosure are operated through a processor and memory. The processor may be composed of one or more processors. In this case, the one or more processors may be general-purpose processors such as CPUs, APs, and DSPs (Digital Signal Processors), graphics-dedicated processors such as GPUs and VPUs (Vision Processing Units), or artificial intelligence-dedicated processors such as NPUs. The one or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, if the one or more processors are artificial intelligence-dedicated processors, the artificial intelligence-dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0344] Figure 36 is a diagram illustrating an offline-online learning method.

[0345] Referring to FIG. 36, the offline-online learning method may include an offline pre-learning step and an online fine-tuning step. Throughout the disclosure, the offline pre-learning step may be referred to as the offline learning step, and the online fine-tuning step may be referred to as the online learning step.

[0346] In one embodiment, during the offline pre-training stage, the policy module (621) and value function module (622) of the agent (620) can be simultaneously trained using transitions (state, action, reward, next state, etc.) included in the offline dataset (610). In this stage, parameters are updated using only the dataset (610) without real-time interaction with the environment, and the pre-trained agent (620) can be obtained.

[0347] In one embodiment, during the online fine-tuning stage, a pre-trained agent (620) interacts with the environment (640) to collect transitions, and the collected transitions can be stored in an online buffer (630). Upon entering the online fine-tuning stage, both the policy module (621) and the value function module (622) are switched to a learning state, and parameters can be updated by mixing the latest transitions accumulated in the online buffer (630) with an offline dataset (610) that is still available for reference as needed. That is, the offline-online learning method corresponding to FIG. 36 is based on a single value function (Q) system and may adopt a configuration in which the same value function learned in the offline stage is continuously updated in the online stage. That is, the offline-online learning method corresponding to FIG. 36 does not include a separate pre-training process by freezing (maintaining) the policy or value function prior to online fine-tuning.

[0348] The offline-online learning method corresponding to Fig. 36 has the advantage of being simple to implement and requiring minimal pipeline changes. However, when the data distribution formed by the offline dataset (610) and the online interaction is different (distribution shift), the value function formed in the offline stage may not accurately reflect the online initial distribution, which may lead to bias in value estimation or increased conservatism. As a result, instability in policy updates or performance degradation may occur during the initial online fine-tuning stage, and a problem has been reported where a small amount of initial transition accumulated in the online buffer (630) is not sufficiently reflected, causing an adaptation delay. In some cases, an agent initialized with offline pre-training may exhibit limited performance advantages compared to random initialization, or may even become disadvantageous.

[0349] Meanwhile, FIG. 36 is illustrated with conceptual simplification of each component for clarity of explanation, and the selection of algorithms (e.g., type of actor-critic, form of loss function, replay strategy, etc.) or module boundaries in actual implementation are not limited to FIG. 36. Additionally, whether the offline dataset (610) is continuously used, the sampling policy of the online buffer (630), the use of the target network, etc., may vary depending on the implementation.

[0350] FIG. 37 is a drawing illustrating an offline-online learning method according to one embodiment of the present disclosure.

[0351] Referring to FIG. 37, the offline-online learning method may include an offline pre-learning stage, an online pre-training stage, and an online fine-tuning stage. Throughout the disclosure, the offline pre-learning stage may be referred to as the offline learning stage, the online pre-training stage as the online pre-learning stage, and the online fine-tuning stage as the online learning stage.

[0352] In one embodiment, the learning procedure may proceed sequentially through an offline pre-learning stage, an online pre-training stage, and an online fine-tuning stage.

[0353] In an offline pre-training step according to one embodiment, the policy module (721) and the first value function module (722) of the agent (720) can be simultaneously trained using the transitions included in the offline dataset (710). The policy pre-trained as a result of the offline pre-training step. and value function Q off-pt can be obtained.

[0354] In the online pre-training phase according to one embodiment, the policy module (721) and the first value function module (722) may be characterized as not being updated and being frozen (maintained). That is, the policy and the first value function may not be updated in the online pre-training phase. Frozen Initial online transitions can be collected by performing limited interaction with the environment (740) using [the method]. The collected transitions can be stored in an online buffer (730). At the same time, the second value function module (723) can be set to a learning state and pre-trained by using the transitions from the offline dataset (710) and the online buffer (730) together. In one embodiment, the learning objective of the second value function may include, but is not limited to, a meta-adapting (inner-outer) form.

[0355] In an online fine-tuning step according to one embodiment, the policy, the first value function, and the second value function may all be transitioned to a learning state. The policy update is a joint value function Q defined as the weighted sum of the two value functions. mix It can be performed in a direction that increases the expected value (or advantage) for. In one example, the joint value function Q mix It can be defined as follows [Mathematical Formula 1].

[0356]

[0357] In one embodiment, as learning progresses, weight k is initially set small to rely more on offline knowledge, and then gradually increased to enhance online adaptability. According to one embodiment of the present disclosure, scheduling may be performed to ensure smooth transitions. Additionally, transitions from the online buffer (730) may be used in conjunction with a balanced replay policy that guarantees a minimum ratio within the mini-batch.

[0358] FIG. 37 is merely a conceptual example for illustrative purposes, and the physical arrangement of modules, data paths, icon displays, etc., may vary depending on the implementation. For example, the specific form of the combined value function (linear, non-linear combination), the schedule of k (linear, exponential, adaptive, etc.), and the sampling policy of the online buffer (double buffer, weighted sampling) may be selectively modified depending on performance or resource constraints.

[0359] A method according to one embodiment is applicable to continuous / discrete action spaces and various actor-critic backbones, and is configured to use only learned policies during inference, so as not to increase operational overhead.

[0360] FIG. 38 is a block diagram of a system according to one embodiment of the present disclosure.

[0361] Referring to FIG. 38, the system (800) comprises a processor (810), memory (820), a transceiver (830), an offline buffer (840), an online buffer (850), a policy module (860), and a first value function (Q off ) module (870), second value function (Q onIt may include a module (880), a k-value scheduler (890), a sampler / replay controller (not shown), etc. However, not all components shown in FIG. 38 are essential components of the system (800). The system (800) may be implemented with more components than those shown in FIG. 38, or with fewer components than those shown in FIG. 38. In addition, the processor (810), memory (820), and transceiver (830) may be implemented in the form of a single chip.

[0362] In one embodiment, the transceiver (830) may communicate with a terminal or other electronic device connected via wired or wireless connection. Various types of data, such as programs and files, such as applications, may be installed and stored in the memory (820). The processor (810) may access and use the data stored in the memory (820), or may store new data in the memory (820). In one embodiment, the system (800) may further include a database (not shown).

[0363] The processor (810) controls the overall operation of the system (800) and may include at least one processor, such as a CPU, GPU, etc. The processor (810) may control other components included in the system (800) to perform operations to operate the system (800). For example, the processor (810) may execute a program stored in memory (820), read a stored file, or save a new file. In one embodiment, the processor (810) may perform operations to operate the system (800) by executing a program stored in memory (820).

[0364] In one embodiment, the offline buffer (840) may store transitions (state, action, reward, next state, etc.) collected offline or provided from an external source. The offline buffer (840) is used for learning the policy module (860) and the first value function module (870) during the offline pre-learning phase and may also be optionally referenced during the online phase.

[0365] In one embodiment, the online buffer (850) can store transitions newly collected through interaction with the environment by the execution of the policy module (860). The online buffer (850) can be sampled according to a balanced replay policy in the online pre-training and online fine-tuning stages to ensure a minimum online rate within the mini-batch.

[0366] In one embodiment, a sampler / replay controller (not shown) may configure mini-batches from an offline buffer (840) and an online buffer (850) and provide them for updates to each module. The sampler / replay controller (not shown) controls replay policies such as sampling rate, priority, prevention of duplication, and sequence length, and may mitigate adaptation delays for distribution shifts by enforcing a minimum rate of online transitions.

[0367] In one embodiment, the first value function module (870) may be trained based on an offline buffer (840) during the offline pre-training phase. The first value function may be frozen during the online pre-training phase and updated again during the online fine-tuning phase. The first value function module (870) may suppress overestimation by including a target network, double Q, clipping, or weighted tampering.

[0368] In one embodiment, the second value function module (880) may be pre-trained using both the offline buffer (840) and the online buffer (850) during the online pre-training phase. In one embodiment, the second value function module (880) may adjust parameters in a manner that minimizes online distribution loss after undergoing internal updates to the offline distribution, including for meta-adaptation (bilevel) purposes. The second value function may also be continuously updated during the online fine-tuning phase.

[0369] In one embodiment, the policy module (860) is trained with an offline buffer (840) during the offline pre-training phase, and the policy may be frozen during the online pre-training phase. In the online fine-tuning phase, the combined value function Q mix = (1-k) Q off + k Q on The policy can be updated based on [this]. At this time, the policy module (860) can use an advantage estimate or a policy gradient.

[0370] In one embodiment, the k-value scheduler (890) may calculate a combined weight k based on learning progress, a distribution difference indicator, or a performance indicator and provide it to the policy module (860). Initially, the k value may be set small to utilize offline knowledge, and as learning progresses, the k value may be increased to expand the weight of online adaptability.

[0371] According to one embodiment, a learning batch can be supplied from an offline / online buffer (840, 850) to a sampler / replay controller, and from the sampler / replay controller to a first and second value function module (870, 880) and a policy module (860).

[0372] In one embodiment, the output of the first and second value function modules (870, 880) is a combined value function (Q) according to the k value provided by the k value scheduler (890). mixPolicy updates can be performed by integrating into ). The transitions obtained as a result of the execution of the policy module (860) are stored in an online buffer (850) and can be reused for subsequent learning.

[0373] The components of this embodiment may be physically implemented in a single device or deployed across multiple computing nodes distributed via a network. Additionally, the boundaries of each module may be integrated or separated depending on the implementation, and the form of the combined value function (e.g., linear, non-linear, etc.), the k-value schedule (linear, exponential, adaptive, etc.), and the replay policy (double buffer, weighted sampling, etc.) may be varied according to performance and resource constraints.

[0374] According to one embodiment of the present disclosure, in order to mitigate value function distortion caused by distribution shifts during the process of performing online reinforcement learning after offline reinforcement learning, a policy learned in the offline stage ( ) and value function(Q off Freeze ) during pre-training, and a second value function (Q) adaptive to the online distribution on After separately pre-training ), Q in the online fine-tuning stage off wa Q on Weighted combination of (Q mix A learning procedure and system that update the policy to ) may be provided. In one embodiment, k is scheduled according to the learning progress, and the utilization of online transitions is ensured through balanced replay, thereby improving convergence speed and final performance.

[0375] FIG. 39 is a flowchart illustrating an offline-online learning method according to one embodiment of the present disclosure.

[0376] Referring to FIG. 39, in operation 910, the processor can learn a first value function by performing offline reinforcement learning based on offline data. In one embodiment, offline reinforcement learning can be learned based on offline data and an offline policy. Throughout the disclosure, the policy may be either a probabilistic or a deterministic policy, and the update may include off-poly actor-criticism, trust-region, KL constraints, entropy normalization, etc.

[0377] In one embodiment, the processor may further learn an offline policy based on offline data. In this case, the learning of the first value function and the learning of the offline policy may be performed simultaneously or alternately.

[0378] In operation 930, the processor may learn a second value function by performing online pre-learning. In one embodiment, the processor may interact with the environment according to the offline policy while freezing (maintaining) the offline policy and the first value function, and accumulate initial online transition data in an online buffer. In one embodiment, when the number of initial transitions reaches a predetermined threshold, the process may proceed to the next step. For example, the pre-threshold may be a quantitative indicator such as 300k or a relative indicator such as 50%.

[0379] In one embodiment, the processor may pre-learn a second value function by using offline data and initial online transition data together. One embodiment may include a meta-adaptive (bilevel) objective to perform one or more virtual updates on the offline data as an inner step, and then, as an outer step, adjust the parameters of the second value function so that the update result has low loss for the online transition. At this time, the offline policy and the first value function may remain frozen (maintained).

[0380] In one embodiment, the processor may acquire online transition data by performing limited interaction with the environment according to the offline policy while freezing (maintaining) the offline policy and the first value function. Additionally, the processor may learn the second value function by performing online pre-learning based on the offline data and the online transition data. At this time, the processor may learn the second value function by performing online pre-learning based on the offline data and the online transition data while freezing (maintaining) the offline policy and the first value function.

[0381] According to one embodiment of the present disclosure, value estimation distortion caused by distribution shift can be mitigated by freezing the policy and value functions learned in the offline phase during online pre-training and separately pre-training a second value function adaptive to the online distribution.

[0382] In addition, according to one embodiment of the present disclosure, by freezing (maintaining) the policy and the first value function during the online pre-training stage, the possibility of negative transfer and fine-tuning failure caused by early modification of existing parameters can be reduced.

[0383] In one embodiment, the first value function and the second value function may share a feature extractor (backbone) or use completely independent parameterization. The initialization of the second value function may use the parameters of the first value function as a seed or use random initialization.

[0384] In operation 950, the processor can perform online reinforcement learning using a third value function generated based on a first value function and a second value function.

[0385] In one embodiment, the third value function may be generated by combining the first value function and the second value function as a weighted sum. That is, the third value function Qmix It can be expressed as (1-k)* (first value function)+k*(second value function). The processor can initialize a weight k to combine the first value function and the second value function. For example, the initial value of k can be from 0.05 to 0.7. After that, the processor can update the policy to increase the expected value (or advantage estimate) of the third value function.

[0386] In one embodiment, the weight (1-k) of the first value function and the weight k of the second value function may be updated according to the extent of online reinforcement learning. That is, the weight (1-k) of the first value function and the weight k of the second value function may be adjusted according to a distribution difference indicator between the offline data distribution and the online data distribution. The distribution difference indicator may include any statistical distance, distance in the embedding space, or a performance gap-based indicator.

[0387] For example, the processor can schedule k to be monotonically non-decreasing based on the training step, performance metrics, or distribution difference metrics (e.g., f-divergence estimate, embedding space distance, return gap). That is, k may have a small value initially to reflect offline confidence, and increase as training progresses. The way k increases can follow linear, exponential, or adaptive patterns, but is not limited to these.

[0388] In one embodiment, the third value function was described as being determined by a linear weighted sum, but it is not limited thereto and can also be implemented as a non-linear combination, a gating network, an attention-based combination, etc.

[0389] For example, if the first time point is earlier than the second time point, the weight (1-k) of the first value function at the first time point is greater than the weight (1-k) of the first value function at the second time point, and the weight (k) of the second value function at the first time point is smaller than the weight (k) of the second value function at the second time point.

[0390] In one embodiment, during online reinforcement learning, the processor [determines] the minimum ratio (r) of online transitions within a minibatch from the offline buffer and the online buffer. on It can be sampled to guarantee ). For example, r on can be set in the range of 0.10 to 0.60. Throughout the entire disclosure, the minimum ratio of online transitions within a minibatch is such that, when B is the total size of the minibatch configured at learning step t, the number of online transitions (transitions collected through actual interaction with the environment) included in the minibatch is at least It can refer to a policy that enforces (guarantees) this. Such minimum ratio enforcement can mitigate adaptation lag to online data distributions that differ from offline data distributions, prevent small amounts of recent transitions from being diluted in the loss contribution early in training, and mitigate bias in policy / value estimation under distribution shifts.

[0391] In one embodiment, the sampler / replay controller Calculate the minimum number by calculating, and depending on the duplicate allowance setting in the online replay buffer, Extract samples, and the remaining number (B- ) can be extracted from the offline buffer. Additionally, the sampler / replay controller can combine the two parts to form the final mini-batch.

[0392] In one embodiment, the sampler / replay controller has a number of sampled result samples In cases where the number is smaller, the insufficient sampling volume can be addressed through methods such as redundant sampling that satisfies the minimum number by allowing duplicates of existing online transitions, loss accumulation correction that uses all available online transitions and records the deficit to compensate for it with additional online weights in subsequent mini-batches, or alternative weighting methods that adjust the contribution of online terms by increasing the loss weights even if the actual number is insufficient.

[0393] In one embodiment, the buffer may use priority replay, and priority application is performed after the minimum ratio is satisfied (i.e., first (Filling the space and sampling within it according to priority). To limit duplication, time intervals, episode variance, state similarity thresholds, etc., can be applied to prevent over-concentration of highly correlated samples.

[0394] In one embodiment, the processor may update the first value function and the second value function, respectively, according to the bootstrapping loss during the progress of online reinforcement learning. The processor may optionally apply one or more of a target network, dual Q, clipping, or weighted tampering for stabilization.

[0395] In one embodiment, when a cumulative reward, success rate, convergence metric, or predefined interaction budget is reached, the processor may terminate online reinforcement learning or generate a parameter snapshot and transition to the inference phase.

[0396] In one embodiment, the initial number of online samples, the minimum rate of online transitions within the minibatch (r on ), k-value schedules, etc., can be adaptively configured according to environmental characteristics, offline data quality, and safety constraints.

[0397] In one embodiment, a policy learned through an online-offline learning procedure may be used to produce an action (decision) output based on the state or observations of the target system in the inference stage. The inference stage may consist of (i) receiving input, (ii) preprocessing, (iii) policy execution, (iv) optional validation and safety gating, (v) control / decision output, (vi) log / transition storage, etc.

[0398] In one embodiment, the input receiving step may include receiving a current state, observations, context, etc. through at least one sensor, gateway, API, etc.

[0399] In one embodiment, the preprocessing step may include a normalization step, an embedding step, a missing value correction step, etc.

[0400] In one embodiment, the policy execution step may include a step of providing input to a learned policy to produce an action or decision result.

[0401] In one embodiment, the validation step may include determining a third value function generated from a first value function and a second value function, calculating a value according to the third value function, and selecting a payoff policy or alternative action based on risk and uncertainty indicators. The processor may select an alternative action or apply additional measurements, heuristics, etc. for double-checking when the third value function is less than a preset value. The preset value may be a fixed value or may be adaptively adjusted according to risk, uncertainty, performance indicators, etc.

[0402] In one embodiment, the control / decision output step may include a step of outputting an actuator control signal, a recommendation result, a ranking generation result, a resource allocation plan, etc.

[0403] In one embodiment, the log / transition storage step may include the step of writing the state, behavior, reward, next state, etc. to an online buffer. This transition may be reused for subsequent online pre-training or online fine-tuning and may be sampled with a balanced replay policy that guarantees a minimum ratio of online transitions within a mini-batch.

[0404] In one embodiment, the processor may acquire a first state and a first observation and determine an output using a learning model in which offline reinforcement learning performed in operation 910, online pre-learning performed in operation 930, and online reinforcement learning performed in operation 950 have all been performed. The processor may determine a first action by inputting the first state and the first observation into the learning model that has completed learning. Additionally, the processor may generate a control signal or provide a decision result based on the first action. The control signal includes driving signals for a robot manipulator, a mobile robot, a drone, a vehicle, a process device, etc., and the decision result may include item recommendation, item ranking, resource allocation, path planning, task scheduling, anomaly detection, etc.

[0405] For example, the processor may receive a state from the sensor, calculate the gripper attitude, speed, force, etc. as a policy, and transmit a control signal to the robot controller to apply a speed limit or safety set projection when the third value function is less than a preset value.

[0406] According to one embodiment of the present disclosure, by using a weighted combination of stable knowledge obtained offline (first value function) and rapid adaptability in online prior learning (second value function), learning instability, performance drop, and policy drift in the initial online fine-tuning stage can be reduced.

[0407] According to one embodiment of the present disclosure, a smooth transition from offline reliability to online adaptability is enabled through k-value scheduling based on learning progress, thereby allowing for gradual improvement in performance without excessive searching or abrupt policy changes.

[0408] According to one embodiment of the present disclosure, by applying balanced replay (guaranteeing a minimum online transition rate within a minibatch), the contribution of the latest interaction data can be secured, thereby reducing adaptation latency and improving sample efficiency.

[0409] According to one embodiment of the present disclosure, the outer objective for the online distribution is optimized after the inner update in the offline distribution through the meta-adaptation (bilevel / inner-outer) objective of the second value function, thereby increasing the convergence speed early in the online phase and reducing overfitting / bias.

[0410] According to one embodiment of the present disclosure, the bias-variance trade-off compared to a single value function system can be favorably adjusted through the separation of roles of the first and second value functions (modular critic design).

[0411] According to one embodiment of the present disclosure, by adaptively adjusting the k-value schedule based on time or distribution difference indicators, the balance between exploration and utilization can be automatically adjusted according to environmental difficulty, offline data quality, and policy movement range.

[0412] According to one embodiment of the present disclosure, the method can be universally coupled to continuous / discrete action spaces and various actor-critic families (e.g., TD3 / SAC / IQL, etc.) backbones, thereby offering high implementation reusability and pluggable extensibility.

[0413] According to one embodiment of the present disclosure, even when the quality of offline data is low or there is a large difference from the online distribution, robustness in an environment of extreme distribution shifts can be secured by setting the initial k value high and increasing the online transition rate.

[0414] According to one embodiment of the present disclosure, additional effects of suppressing overestimation and stabilizing training can be obtained by selectively applying a target network, dual Q, clipping / weighted tempering, entropy normalization, etc.

[0415] According to one embodiment of the present disclosure, the number of initial interactions required in online pre-training ( Since the prior learning effect of the second value function can be secured even when limiting ), operational risk can be reduced in applications with significant safety and cost constraints (robotics, process control, recommendation, etc.).

[0416] According to one embodiment of the present disclosure, the combination of two value functions is utilized only internally during learning, and can be configured to use only the learned policy during inference, thereby realizing learning benefits without increasing overhead during operation.

[0417] According to one embodiment of the present disclosure, quantitative performance indicators (e.g., cumulative reward, IQM, success rate, etc.) can be improved under the same online interaction budget, and variance between seeds can be reduced, thereby improving reproducibility and predictability.

[0418] According to one embodiment of the present disclosure, the method has a smaller scope of implementation change compared to existing offline-only or purely online methods and can be easily integrated into existing pipelines, resulting in high cost-effectiveness in both research and industrial environments.

[0419] One embodiment of the present disclosure may also be implemented in the form of a recording medium comprising computer-executable instructions, such as program modules executed by a computer. A computer-readable medium may be any available medium accessible by a computer and includes both volatile and non-volatile media, and both removable and non-removable media. Additionally, a computer-readable medium may include both computer storage media and communication media. A computer storage medium includes both volatile and non-volatile, removable and non-removable media implemented by any method or technique for storing information, such as computer-readable instructions, data structures, program modules, or other data. A communication medium typically includes computer-readable instructions, data structures, or program modules and includes any information transmission medium.

[0420] The foregoing description of the present disclosure is for illustrative purposes only, and those skilled in the art will understand that modifications can be easily made to other specific forms without altering the technical spirit or essential features of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0421] The scope of the present disclosure is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts thereof should be interpreted as being included within the scope of the present disclosure.

Claims

1. In the system, One or more processors; and The system includes one or more memories that collectively store instructions that cause the system to perform operations when executed by the above-mentioned one or more processors, and the operations are: The operation of learning a first value function by performing offline reinforcement learning based on offline data; The operation of learning a second value function by performing online pre-learning; and A system comprising the operation of performing online reinforcement learning using a third value function generated based on the first value function and the second value function.

2. In paragraph 1, the third value function is, A system generated by combining the above-mentioned first value function and the above-mentioned second value function as a weighted sum.

3. In paragraph 2, the weights of the first value function and the second value function are, A system characterized by being updated according to the degree of performance of the above-mentioned online reinforcement learning.

4. In paragraph 2, the weights of the first value function and the second value function are, A system that is adjusted according to an indicator of the difference in distribution between offline data distribution and online data distribution.

5. In Paragraph 1, The weight of the first value function at the first time point is greater than the weight of the first value function at the second time point, and the weight of the second value function at the first time point is smaller than the weight of the second value function at the second time point, and The above first point in time is a system that is earlier than the above second point in time.

6. In Paragraph 1, The above offline reinforcement learning is, It is learned based on the above offline data and offline policies, The operation of learning the second value function by performing the above-mentioned online pre-learning is, An operation to acquire online transition data by performing limited interaction with the environment according to the offline policy while maintaining the offline policy; and A system comprising the operation of learning a second value function by performing the online pre-learning based on the offline data and the online transfer data.

7. In Paragraph 1, The above offline reinforcement learning is, It is learned based on the above offline data and offline policies, The operation of learning the second value function by performing the above-mentioned online pre-learning is, An operation to acquire online transition data by performing limited interaction with the environment according to the offline policy while maintaining the offline policy and the first value function; and A system comprising the operation of learning a second value function by performing the online pre-learning based on the offline data and the online transition data while maintaining the offline policy and the first value function.

8. In paragraph 1, the above operations are: An operation to acquire a first state and a first observation; A system comprising the above offline reinforcement learning, the above online pre-learning, and the operation of determining a first action by inputting the above first state and the above first observation to a learning model on which the above online reinforcement learning is performed.

9. In paragraph 8, the above operations are, Based on the first action above, it further includes an action of generating a control signal or providing a decision result, and The above control signal is, It includes a driving signal of at least one of a robot manipulator, a mobile robot, a drone, a vehicle, or a process device, and The above decision result is, A system comprising at least one of item recommendation, item ranking, resource allocation, path planning, task scheduling, or anomaly detection.

10. An operation of learning a first value function by performing offline reinforcement learning based on offline data; The operation of learning a second value function by performing online pre-learning; and A method comprising the operation of performing online reinforcement learning using a third value function generated based on the first value function and the second value function.

11. In paragraph 10, the above third value function is, A method generated by combining the first value function and the second value function as a weighted sum.

12. In paragraph 11, the weights of the first value function and the second value function are, A method characterized by being updated according to the degree of performance of the above-mentioned online reinforcement learning.

13. In paragraph 11, the weights of the first value function and the second value function are, A method that is adjusted according to an indicator of the difference in distribution between offline data distribution and online data distribution.

14. In Paragraph 10, the above third value function is, The weight of the first value function at the first time point is greater than the weight of the first value function at the second time point, and the weight of the second value function at the first time point is smaller than the weight of the second value function at the second time point, and A method in which the first point in time is a point in time earlier than the second point in time.

15. In Paragraph 10, the above-mentioned offline reinforcement learning is, It is learned based on the above offline data and offline policies, The operation of learning the second value function by performing the above-mentioned online pre-learning is, An operation to acquire online transition data by performing limited interaction with the environment according to the offline policy while maintaining the offline policy; and A method comprising the operation of learning a second value function by performing the online pre-learning based on the offline data and the online transfer data.

16. In Paragraph 10, the above-mentioned offline reinforcement learning is, It is learned based on the above offline data and offline policies, The operation of learning the second value function by performing the above-mentioned online pre-learning is, An operation to acquire online transition data by performing limited interaction with the environment according to the offline policy while maintaining the offline policy and the first value function; and A method comprising the operation of learning a second value function by performing online pre-learning based on the offline data and the online transition data while maintaining the offline policy and the first value function.

17. In Paragraph 10, the above operations are, Based on the first action above, it further includes an action of generating a control signal or providing a decision result, and The above control signal is, It includes a driving signal of at least one of a robot manipulator, a mobile robot, a drone, a vehicle, or a process device, and The above decision result is, A method comprising at least one of item recommendation, item ranking, resource allocation, path planning, task scheduling, or anomaly detection.

18. In paragraph 17, the above method is, Based on the first action above, it further includes an action of generating a control signal or providing a decision result, and The above control signal is, It includes a driving signal of at least one of a robot manipulator, a mobile robot, a drone, a vehicle, or a process device, and The above decision result is, A method comprising at least one of item recommendation, item ranking, resource allocation, path planning, task scheduling, or anomaly detection.

19. A program stored on a computer-readable recording medium to execute the method of any one of paragraphs 10 through 18 on a computer.

Images