Apparatus and Method for Generating Policies for SLM
The method and device generate SLM policies by creating a basis dataset and learning through an embodied knowledge graph, addressing computational challenges and environmental complexity, enabling effective operation in limited-capacity devices.
Patent Information
- Application Number
- KR1020240102441
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2044-08-01
AI Technical Summary
Existing technologies face challenges in generating effective inference and planning policies for Small Language Models (SLM) due to high computational requirements of Large Language Models (LLM), especially in devices with limited capacity, and the complexity of adapting to dynamic environments.
A method and device for generating SLM policies that involve creating a basis dataset, learning inference and planning policies through an embodied knowledge graph, and distilling LLM reasoning into a compact model using a two-level hierarchical structure, incorporating Markov Decision Process (MDP) functions and Chain-of-Thought Prompting (CoT) schemes.
Enables the generation of SLM policies that can operate effectively in limited-capacity devices by simplifying reasoning processes, allowing agents to understand and interact with complex and dynamic environments, maintaining performance similar to LLMs.
Smart Images

Figure 112024083878318-PAT00133_ABST
Abstract
Description
Technology Field
[0001] The present invention relates to a generating device and a generating method for generating inference policies and planning policies for SLM. Background Technology
[0002] With the commercialization of artificial intelligence, various studies are underway to derive policies based on specialized datasets. In particular, since operating Large Language Models (LLM) on off-the-shelf devices with low computational power, such as portable terminals, is difficult, various studies are being conducted on the operation of Small Language Models (SLM).
[0003] In addition, significant advancements have been made in applying LLM to work planning in AI. For example, active research is underway to combine the reasoning capabilities of LLM with reinforcement learning (RL)-based fitness models to interpret work instructions and derive actionable robotic skills in the environment, as well as to explore ways to ground LLM in the environment through prompts based on sensory data, reference trajectories, and available skills. Furthermore, research is also being conducted to extend the specific reasoning capabilities of LLM to multimodal data, such as visual observations.
[0004] However, this approach may face practical limitations in making short-term decisions by continuously relying on LLMs. This is particularly true when decision agents must operate on commercial devices with limited capacity. The high computational requirements of LLMs present a significant technical challenge in such scenarios, and active research is currently underway to address this issue.
[0005] Directly distilling LLMs end-to-end into smaller, more resource-efficient models may appear simple, but it may not be effective for complex specific tasks. Research aimed at addressing this technical challenge requires a deep understanding of specific task functions, as this inherently demands long-term multi-step reasoning and the ability to adapt to environmental contexts that change over time. Specific agents frequently encounter new environmental information through interactions with their surroundings. Continuous exposure to these diverse environmental conditions adds complexity and variability, complicating the distillation process and hindering progress in related research. Prior art literature
[0006] Deciphering Political Entity Sentiment in News with Large Language Models: Zero-Shot and Few-Shot Strategies (Alapan Kuila et al. 1, https: / / doi.org / 10.48550 / arXiv.2404.04361, 2024.04.05.) The problem to be solved
[0007] A policy generation device and a generation method for an SLM that learn and generate inference policies and planning policies by generating and verifying a basis data set are provided. means of solving the problem
[0008] One embodiment of a method for generating a policy for SLM may include the steps of receiving a professional dataset, generating a grounds dataset based on the professional dataset and a previously stored initial grounds set, verifying the grounds dataset through a self-validation function, learning an inference policy through an embodied knowledge graph based on the verified grounds dataset, learning a planning policy based on the grounds set of the learned inference policy and the planning policy reconstruction loss, and generating an SLM policy including a final inference policy and a planning policy based on the learned inference policy and planning policy.
[0009] In addition, according to one embodiment, a method for generating a policy for an SLM may be characterized in that, in the step of learning the planning policy, the basis set of the inferred inference policy is generated based on the encoder policy based on the encoder prompt pool, the decoder policy based on the decoder prompt pool, and the attention module while performing the inference policy learning step.
[0010] In addition, according to one embodiment, a method for generating a policy for an SLM may be characterized in that the encoder prompt pool includes a prefix prompt and a postfix prompt.
[0011] In addition, according to one embodiment, a policy generation method for SLM may be characterized in that the set of grounds for the inferred inference policy is generated through grounds reconstruction loss based on a graph extracted through grounds and a knowledge graph search function.
[0012] In addition, according to one embodiment, a policy generation method for SLM, wherein the basis reconstruction loss is a mathematical formula
[0013]
[0014] (Here, :Reconstruction loss, o: Observation, h: Task description, R: Set of grounds, : Basis, g: Graph extracted through the knowledge graph search function)
[0015] It can be characterized as being.
[0016] In addition, according to one embodiment, a method for generating a policy for an SLM may be characterized in that, in the step of learning the inference policy, the embodied knowledge graph is a prompted knowledge graph.
[0017] Additionally, according to one embodiment, a policy generation method for an SLM may be characterized in that the prompted knowledge graph is based on a batch sample comprising a positive pair, which is an embodied knowledge graph executing the same plan, and a negative pair, which is a successive plan step.
[0018] In addition, according to one embodiment, a policy generation method for SLM is characterized in that the prompted knowledge graph is based on contrast learning loss, and the contrast learning loss is a mathematical formula
[0019]
[0020] (Here, :Contrast learning loss, : Batch sample, : embedding space, d: embedding space The sum of distance metrics corresponding to the elements of each ground embedding sequence within ∈Z, : It can be characterized as a margin parameter.
[0021] In addition, according to one embodiment, a policy generation method for SLM, in the step of learning the planning policy, wherein the planning policy is learned by predicting the next plan (a) based on the basis set of the learned inference policy, and a mathematical formula
[0022]
[0023] (Here, :Planned policy, R: Basis set of inferred policy, : It can be characterized as being learned through an inference policy, g: a graph extracted through a knowledge graph search function, and a: the next plan.
[0024] In addition, according to one embodiment, a policy generation method for SLM, wherein the planning policy reconfiguration loss is a mathematical formula
[0025]
[0026] (Here, : Planning policy restructuring loss, o: observation, h: task description, R: basis set, : Planning policy, a: can be characterized as the next plan)
[0027] An embodiment of a policy generation device for SLM includes an input / output interface that receives a professional dataset, a processor that generates an SLM policy based on the professional dataset, and a communication unit that transmits the generated SLM policy to a terminal. The processor generates a basis dataset based on the professional dataset and a pre-stored initial basis set, verifies the basis dataset through a self-verification function, learns an inference policy through an embodied knowledge graph based on the verified basis dataset, learns a planning policy based on the basis set of the learned inference policy and the planning policy reconstruction loss, and generates an SLM policy including a final inference policy and a planning policy based on the learned inference policy and planning policy.
[0028] Additionally, according to one embodiment, the policy generation device for SLM may be characterized in that, when the processor learns the planning policy, the basis set of the inferred inference policy is generated based on an encoder policy based on an encoder prompt pool, a decoder policy based on a decoder prompt pool, and an attention module while performing the inference policy learning step, and the encoder prompt pool includes a prefix prompt and a postfix prompt.
[0029] Additionally, according to one embodiment, the policy generation device for SLM may be characterized in that the processor generates the basis set of the inferred policy through a basis reconstruction loss based on a graph extracted through a basis and knowledge graph search function.
[0030] In addition, according to one embodiment, the policy generation device for SLM, the processor,
[0031] The above-mentioned basis reconstruction loss is a mathematical formula
[0032]
[0033] (Here, :Reconstruction loss, o: Observation, h: Task description, R: Set of grounds, : basis, g: graph extracted through a knowledge graph search function) can be characterized as being a basis.
[0034] Additionally, according to one embodiment, the policy generation device for SLM may be characterized in that, when the processor learns the inference policy, the embodied knowledge graph is a prompted knowledge graph.
[0035] Additionally, according to one embodiment, a policy generation device for an SLM may be characterized in that the processor is based on a batch sample comprising a positive pair, which is an embodied knowledge graph executing the same plan, and a negative pair, which is a successive plan step.
[0036] In addition, according to one embodiment, a policy generation device for an SLM is characterized in that the processor, the prompted knowledge graph is based on a contrast learning loss, and the contrast learning loss is a mathematical formula
[0037]
[0038] (Here, :Contrast learning loss, : Batch sample, : embedding space, d: embedding space The sum of distance metrics corresponding to the elements of each ground embedding sequence within ∈Z, : It can be characterized as a margin parameter.
[0039] In addition, according to one embodiment, the policy generation device for SLM, wherein the processor, when learning the planning policy, learns the planning policy by predicting the next plan (a) based on the basis set of the learned inference policy, and mathematical formula
[0040]
[0041] (Here, :Planned policy, R: Basis set of inferred policy, : It can be characterized as being learned through an inference policy, g: a graph extracted through a knowledge graph search function, and a: the next plan.
[0042] In addition, according to one embodiment, the policy generation device for SLM, the processor, wherein the planning policy reconstruction loss is a mathematical formula
[0043]
[0044] (Here, : Planning policy restructuring loss, o: observation, h: task description, R: basis set, : Planning policy, a: can be characterized as the next plan)
[0045] An embodiment of a central server for generating a policy for an SLM includes an input / output interface for receiving a professional dataset, a processor for generating an SLM policy based on the professional dataset, and a communication unit for transmitting the generated SLM policy to a terminal. The processor generates a basis dataset based on the professional dataset and a pre-stored initial basis set, verifies the basis dataset through a self-verification function, learns an inference policy through an embodied knowledge graph based on the verified basis dataset, learns a planning policy based on the basis set of the learned inference policy and the planning policy reconstruction loss, and generates an SLM policy including a final inference policy and a planning policy based on the learned inference policy and planning policy. Effects of the invention
[0046] According to the policy generation device and generation method for the SLM described above, inference policies and planning policies can be learned and generated by generating and verifying a basis data set based on a professional data set and a basis set. Brief explanation of the drawing
[0047] FIG. 1 is a block diagram of a policy generation device for an SLM according to one embodiment. FIG. 2 is a block diagram of a central server according to one embodiment. FIG. 3 is a block diagram of a processor and an input / output interface unit according to one embodiment. FIG. 4 is a block diagram of a processor and an input / output interface unit according to another embodiment. FIG. 5 is a conceptual diagram illustrating the relationship between a query and a basis according to one embodiment. FIG. 6 is a conceptual diagram of a policy learning algorithm according to one embodiment. FIG. 7 is a flowchart of a policy generation method for SLM according to one embodiment. FIG. 8 is a flowchart of a policy generation method for SLM according to another embodiment. FIG. 9 is a flowchart of a policy generation method for SLM according to another embodiment. Specific details for implementing the invention
[0048] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims.
[0049] The terms used in this specification will be briefly explained, and the invention will be described in detail.
[0050] The terms used in this invention have been selected based on currently widely used general terms, taking into account their functions within the invention; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this invention should be defined not merely by their names, but based on their meanings and the overall content of the invention.
[0051] Throughout the specification, when a part is described as "comprising" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as "part," "module," and "unit" used in the specification refer to a unit that processes at least one function or operation and may be implemented as software, hardware components such as FPGAs or ASICs, or a combination of software and hardware. However, the terms "part," "module," and "unit" are not limited to software or hardware. "Part," "module," and "unit" may be configured to reside in an addressable storage medium or configured to run one or more processors. Accordingly, as an example, terms such as "part," "module," and "unit" include components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.
[0052] Below, embodiments of the present invention are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the invention. Additionally, parts of the drawings that are irrelevant to the description are omitted to clearly explain the invention.
[0053] Terms including ordinal numbers, such as "first," "second," etc., may be used to describe various components, but the components are not limited by the terms. The terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of multiple related items or any one of the multiple related items.
[0054] Hereinafter, with reference to the attached drawings, an embodiment of a method for generating a policy for SLM, a device for generating a policy for SLM, and a central server for generating a policy for SLM will be described.
[0055] Hereinafter, with reference to FIGS. 1 to 6, an embodiment of a policy generation device for SLM and a central server that generates policies for SLM will be described.
[0056] FIG. 1 is a block diagram of a policy generation device (1) for an SLM according to one embodiment, FIG. 2 is a block diagram of a central server according to one embodiment, FIG. 3 is a block diagram of a processor (110) and an input / output interface (130) according to one embodiment, and FIG. 4 is a block diagram of a processor (110) and an input / output interface (130) according to another embodiment.
[0057] Referring to FIG. 1, a policy generating device (1) for SLM may include a central server (100) and a terminal (200).
[0058] The terminal (200) can receive a specialized dataset for generating an SLM policy from a user, and the terminal (200) may include a mobile terminal (210), a computing terminal (220), a workstation (230), and an agent server (240).
[0059] The central server (100) can generate a source dataset based on a professional dataset and a pre-stored initial source set, and can verify the generated source dataset through a self-validation function. The central server (100) can learn an inference policy through an embodied Knowledge Graph (KG) based on the verified source dataset, and can learn a planning policy based on the source set of the learned inference policy and the planning policy reconstruction loss. The central server (100) can generate an SLM policy including a final inference policy and a planning policy based on the learned inference policy and planning policy.
[0060] The policy generation device (1) and central server (100) for the SLM of the present invention can minimize divergence from the distribution of policies (LLM) in LLM (large language models)-based policy distillation and explore a unique two-level hierarchical structure in the decomposition and distillation of the inference ability of LLM.
[0061] In Reinforcement Learning (RL), the environment of an embodied agent is modeled as a Partially Observable Markov Decision Process (POMDP), which can be represented as a tuple of (S, A, P, G, H, R, Ω, O). Here, s∈S is the state space, a∈A is the action space, P : S Х A Х S → [0,1] is the transition probability, G∈G is the goal space, h∈H is the high-level task description, and R : S Х A Х G → R is the reward function.
[0062] A distinct aspect of the environment of an embodied agent lies in the nature of partial observation, which can be characterized by an observation space of o∈Ω and a conditional observation probability O : S Х A → Ω. This explains the agent's limited perception, which can make decision-making complex and reflect real-world situations.
[0063] The present invention provides a powerful Small Language Model (SLM) based policy that can be used in a commercial device with limited capacity, such as a terminal (200). ...can be achieved. This is an LLM-based policy It may be similar to the performance demonstrated in this embodied job plan. The SLM policy may include a final inference policy and a planning policy, and the SLM policy is It can be derived from mathematical formula 1.
[0064]
[0065] Here, D is a distance function such as the Kullback-Leibler divergence, and is an environmental discounting factor.
[0066] For embodied tasks, it is important for the agent to possess reasoning capabilities that enable it to understand and interact with complex and dynamic environments. However, when using SLM-based policies, it is necessary to simplify the reasoning process due to the limited capacity of the model. This can be achieved by integrating MDP functions specified by reinforcement learning (RL) formulas—such as goals, states, observations, actions, residual rewards, and sub-goals—into the reasoning process.
[0067] The policy generation device (1) and central server (100) for SLM of the present invention refer to this type of environmental information and MDP function as rationale, which can function as justification or hint to help explain the reasoning behind the plan. The policy generation device (1) and central server (100) for SLM can achieve an SLM-based policy by utilizing this rationale to effectively distill the embodied reasoning ability of LLM into a compact model. The present invention relates to a framework for a policy for SLM, which can 1) generate and validate a rationale dataset, 2) learn (distill) and generate an SLM policy (reasoning policy and planning policy) through an embodied knowledge graph (Embodied KG), and 3) evaluate the SLM policy in an environment where it is first experienced in a zero-shot deployment.
[0068] Specifically, in the evidence dataset construction (generation and validation) phase, a Chain-of-Thought Prompting (CoT) scheme can be utilized using LLM to extract evidence from expert dataset transfers (e.g., a series of action plans) in the environment. This is achieved by using RL-specific queries as prompts through in-context learning equipped with Markov Decision Process (MDP) capabilities. In the next phase, SLM policy distillation (inference policy and planning policy), a two-level hierarchical SLM-based policy is established based on the embodied knowledge graph. This includes an inference policy trained to generate evidence via a single-level CoT optimized by behavior-based contrastive learning, and a learned planning policy that infers action plans via CoT prompts using this evidence as a guide. In the zero-shot deployment phase, the distilled SLM policy can be evaluated in a zero-shot manner in a new environment where the task description, object locations, and indoor scenes have changed.
[0069] In embodied tasks, it is essential for an agent to possess reasoning capabilities that enable it to understand and interact with complex and dynamic environments. However, when using SLM-based policies, it is particularly necessary to simplify the reasoning process due to limitations in model capacity. This can be achieved by integrating MDP functions—such as goals, states, observations, actions, remaining rewards, and sub-goals specified by reinforcement learning (RL) formulas—into the reasoning process. In this invention, these environmental information and MDP functions are defined as rationales, which can function as justifications or hints that help explain the reasoning behind a plan. By utilizing these rationales, this invention achieves SLM-based policies by effectively distilling the embodied reasoning capabilities of LLM into a compact model.
[0070] The central server (100) may include a processor (110), a communication unit (120), and an input / output interface (130).
[0071] The communication unit (120) can receive a user input text dataset from the terminal (200). The communication unit (120) can be implemented, for example, using at least one communication module (e.g., a LAN card, a short-range communication module, or a mobile communication module).
[0072] The input / output interface (130) can provide the user with the ability to directly input the professional dataset to the central server (100) without the communication unit (120) receiving the professional dataset from the terminal (200). The input / output interface (130) may include an input unit and an output unit.
[0073] The input / output interface (130) may be in the form of a push button for pressing operation buttons, or it may operate the operation of the policy generating device (1) for SLM and the central server (100) for generating policies for SLM that the user wants, such as a slide switch, or it may input the operation the user wants in a touch format. In addition, various types of input devices for inputting the operation of the policy generating device (1) for SLM and the central server (100) for generating policies for SLM that the user wants may be used as examples of input units.
[0074] For example, the input / output interface (130) may include a display. The display may be provided as a Cathode Ray Tube (CRT), Digital Light Processing (DLP) panel, Plasma Display Panel, Liquid Crystal Display (LCD) panel, Electro Luminescence (EL) panel, Electrophoretic Display (EPD) panel, Electrochromic Display (ECD) panel, Light Emitting Diode (LED) panel, or Organic Light Emitting Diode (OLED) panel, but is not limited thereto. Additionally, the output unit may include a Central Processing Unit (CPU) implemented as a microprocessor, a Graphic Processing Unit (GPU), and various types of storage devices, and such devices may be provided on an embedded Printed Circuit Board (PCB).
[0075] The processor (110) can generate a source dataset based on a professional dataset and a pre-stored initial source set, and can verify the generated source dataset through a self-validation function. The central server (100) can learn an inference policy through an embodied Knowledge Graph (KG) based on the verified source dataset, and learn a planning policy based on the source set of the learned inference policy and the planning policy reconstruction loss. The central server (100) can generate an SLM policy including a final inference policy and a planning policy based on the learned inference policy and planning policy.
[0076] The processor (110) may include, for example, a central processing unit (CPU), a graphic processing unit (GPU), a micro controller unit (MCU), an application processor (AP), an electronic control unit (ECU), and / or at least one electronic device capable of performing various calculations and control processing. These devices may be implemented, for example, by using one or more semiconductor chips, circuits, or related components alone or in combination.
[0077] The processor (110) may include a basis data generation unit (111), a policy learning unit (112), a policy generation unit (113), and a memory (114).
[0078] The evidence data generation unit (111) can generate an evidence dataset based on a professional dataset and a pre-stored initial evidence set. The evidence data generation unit (111) can verify whether the generated evidence dataset matches the action plan by using LLM as a self-verification function.
[0079] Specifically, the basis data generation unit (111) is a professional data set ( Can receive ). is each transition, is observation, is an action (plan), means a high-level task description. The basis data generation unit (111) is an expert dataset By extending the basis dataset ( It can generate ). Here, each transition is the basis set It can be supplemented by. To obtain a set of grounds configured for a given embodied task, the grounds data generation unit (111) integrates in-context learning with MDP capabilities and a CoT prompt mechanism. This can be performed iteratively using a series of RL-specific queries from the LLM as prompts. Subsequently, the grounds set undergoes evaluation by the LLM to form a grounds dataset. It is integrated into.
[0080] The evidence data generation unit (111) can perform in-context learning equipped with an MDP function. Specifically, the evidence data generation unit (111) can perform in-context learning that continuously updates examples in the evidence dataset to extract evidence from LLM using a transfer τ. The evidence data generation unit (111) is a search function The source data generation unit (111) can use a search function to find the transition τ in the source dataset and the set of tuples ( Take ) as input, search C for the top k semantically related tuples for a given τ, and the example set ...can be obtained. Semantic relevance can be calculated as the inner product between the transitions τ and c through a pre-trained language embedding model E. That is, the relevance score is It can be obtained as.
[0081] The basis data generation unit (111) uses a set of examples to generate a predefined series of RL-specific queries LLM policy together with By providing a prompt to ), a set of grounds can be generated sequentially as in Equation 2.
[0082]
[0083] Here silver It represents a set of grounds generated earlier. In this process It enhances LLM's in-context learning to query It enables effective response to. In particular, RL-specific queries can extract MDP functions required for embodied work planning, such as goals, states, plans, observations, plan history, and sub-goals. Examples of such queries and rationale are illustrated in Fig. 5.
[0084] The evidence data generation unit (111) can verify the evidence dataset by using LLM as a self-criticism function. Specifically, the evidence data generation unit (111) uses LLM as a self-criticism function to ensure that the evidence set R matches action plan a. The evidence data generation unit (111) provides a prompt to the LLM to check whether plan a can be derived solely from the evidence set. It uses. If the basis set does not provide sufficient information about the plan, the basis data generation unit (111) may start again from searching for in-context examples. Otherwise, the basis data generation unit (111) uses a newly generated tuple ( ) can be integrated into the evidence dataset. The evidence data generation unit (111) can collect evidence containing sufficient information to induce a plan in expert transfer through self-verification. The operation by which the evidence data generation unit (111) verifies the evidence dataset is as shown in Equation 3.
[0085]
[0086] The policy learning unit (112) can generate and learn inference policies and planning policies through an embodied knowledge graph based on a verified evidence dataset.
[0087] The policy learning unit (112) uses the basis dataset to develop the inference ability of LLM into an SLM-based policy ( To distill it into ), the policy is organized into a two-level hierarchical structure. The first level is the inference policy ( With ), given observation( ), Job Description( ), embodied knowledge graph( The basis set is inferred from ). The second step is the planning policy ( ...as ), a planning policy is generated based on the rationale generated from the inference policy. The process of distilling the SLM-based policy is as shown in Equation 4.
[0088]
[0089] The embodied knowledge graph is an internal component of an SLM-based policy that encapsulates environmental information. During the training process, fine-tuning using soft prompts can be employed to adopt the SLM-based policy. This is effective for adopting SLMs with limited reasoning capabilities.
[0090] Since the agent can continuously interact with the environment and accumulate information for task completion, it is important to efficiently represent the information to prompt an SLM-based policy. The policy learning unit (112) uses an embodied knowledge graph to create a triplet set ( Includes ). Here is the subject, is a relationship, represents an object. For example, "an apple is on the table" and "an agent picks up a knife" are expressed as triplets of "Apple-On-Table" and "Agent-Pickup-Knife". The policy learning unit (112) updates the embodied knowledge graph as in Equation 5 through the update function (U) at each planning step t.
[0091]
[0092] To prompt an SLM-based policy, the policy learning unit (112) can use a knowledge graph search function (V) that searches in triplets (g) related to observations (o) and task descriptions (h) as in Equation 6.
[0093]
[0094] The associated triplet is selected by a pre-trained semantic relevance function (S) between each triplet and the observation and task description, where δ is a threshold hyperparameter and g represents a graph extracted through a knowledge graph search function.
[0095] The data learning department, in relation to inference policy distillation, inference policy ( ) can generate a set of grounds from a given observation (o), task description (h), and embodied knowledge graph (g) as in Equation 7. The data learning unit uses an encoder-decoder architecture and an attention module.
[0096]
[0097] To generate evidence through a single-stage CoT, the data learning unit uses a soft prompt pool ( Uses ). Encoder policy( ) is a prefix prompt, which is two prompt pools ( ) and postfix prompt( It includes ) and is derived as in mathematical formula 8.
[0098]
[0099] Each prefix prompt ( ) is a query( Initialized based on the language embedding of ), and each postfix prompt( ) is initialized randomly. In addition, to highlight information from each evidence and deliver it sequentially in a manner consistent with the construction of the evidence dataset, the attention module (as shown in Equation 9) ) is causal attention( ) and gate attention( It may include ).
[0100]
[0101] Here, α is a scaling factor that regulates the output of the attention mechanism. Decoder policy ( ) is the decoder prompt pool as in mathematical formula 10( You can generate a grounds set (R) using ).
[0102]
[0103] The data learning unit uses the update function (U) and the knowledge graph search function (V) as in Equation 11 to obtain the evidence dataset ( Along with the embodied knowledge graph generated in ), the inference policy can be optimized through the evidence reconstruction loss.
[0104]
[0105] This loss is based on each basis ( It is calculated by summing the log-likelihoods of the probability of generating ).
[0106] Considering that minute changes in the environment can have an inconsistent impact on the agent's plan, the data learning unit may include a prompted knowledge graph representation for causal and gate attention using behavior-based contrastive learning. The prompted knowledge graph representation enables single-step inference on multiple grounds through grounding policies. The data learning unit uses batch samples ( Samples ). Here means a positive pair, represents a negative pair. Specifically, a positive pair consists of an embodied knowledge graph executing the same plan, and a negative pair is defined as a successive plan step. Subsequently, the contrastive learning loss can be calculated as shown in Equation 12.
[0107]
[0108] Here and d is the embedding space It means the sum of distance metrics corresponding to the elements of each ground embedding sequence within, and represents the margin parameter.
[0109] The data learning unit can distill planning policies. Planning policies ( ) is an inference policy( Based on the set of grounds generated in ), the next plan (a) is predicted as in Equation 13.
[0110]
[0111] The data learning unit can optimize the planning policy through reconstruction loss as shown in Equation 14.
[0112]
[0113] The data learning unit performs a policy distillation procedure with an algorithm as shown in Fig. 6, where the losses of (11), (12) and (14) are used for the inference policy and the planning policy, respectively.
[0114] The policy generation unit (113) can generate an SLM policy including a final inference policy and a planning policy based on the inference policy and planning policy learned in the policy learning unit (112).
[0115] Memory (114) can store data required for computation in a policy generation device for SLM. Memory (114) can store a professional dataset, a pre-stored initial set of grounds, a set of grounds for inference policies, a self-validation function, a generated and validated grounds dataset, an embodied knowledge graph, a prompt knowledge graph, learned inference policy data, planning policy reconstruction loss, grounds reconstruction loss, contrast learning loss, final inference policy, and final planning policy.
[0116] The memory (114) may include at least one of a main memory and an auxiliary memory. The main memory may be implemented using a semiconductor storage medium such as ROM and / or RAM, for example, and the auxiliary memory may be implemented based on a device capable of storing data permanently or semi-permanently, such as a flash memory device (Solid State Drive (SSD), etc.), an SD (Secure Digital) card, a hard disk drive (HDD), a compact disc, a DVD, or a laser disc.
[0117] Hereinafter, an example of a policy generation method for SLM will be described with reference to FIGS. 7 to 9.
[0118] FIG. 7 is a flowchart of a method for generating a policy for an SLM according to one embodiment, FIG. 8 is a flowchart of a method for generating a policy for an SLM according to another embodiment, and FIG. 9 is a flowchart of a method for generating a policy for an SLM according to yet another embodiment.
[0119] According to one embodiment of a method for generating a policy for SLM, a processor may first generate a source dataset (S100) based on a received professional dataset and a pre-stored initial source set, and verify the source dataset (S200) through a self-verification function. The processor may learn an inference policy (S300) through an embodied knowledge graph (Embodied KG) based on the source dataset verified by the processor, and learn a planning policy (S400) based on the source set of the learned inference policy and the planning policy reconstruction loss. Subsequently, the processor may generate an SLM policy (S500) including a final inference policy and a planning policy based on the learned inference policy and planning policy.
[0120] According to another embodiment of the method for generating a policy for SLM, the processor may first generate a source dataset (S100) based on a received professional dataset and a pre-stored initial source set, and verify the source dataset (S200) through a self-verification function. The processor may learn an inference policy through an embodied knowledge graph (Embodied KG) based on the source dataset verified by the processor (S300), and generate a source set of the learned inference policy through a source reconstruction loss based on a graph extracted through a knowledge graph search function (S310). Additionally, the processor may learn a planning policy (S400) based on the source set of the learned inference policy and the planning policy reconstruction loss. Subsequently, the processor may generate an SLM policy (S500) including the final inference policy and planning policy based on the learned inference policy and planning policy.
[0121] According to another embodiment of the method for generating a policy for SLM, the processor may first generate a basis dataset (S100) based on a received professional dataset and a pre-stored initial basis set, and verify the basis dataset (S200) through a self-verification function. The processor may learn an inference policy (S305) through a Prompted Knowledge Graph (Prompted KG) based on the basis dataset verified by the processor, and learn a planning policy (S400) based on the basis set of the learned inference policy and the planning policy reconstruction loss. Subsequently, the processor may generate an SLM policy (S500) including a final inference policy and a planning policy based on the learned inference policy and planning policy.
[0122] Those skilled in the art related to the embodiments of the present invention will understand that they may be implemented in modified forms without departing from the essential nature of the description. Therefore, the disclosed methods should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is defined by the claims, not by the detailed description of the invention, and all variations within the scope of the claims should be interpreted as being included within the scope of the present invention.
[0123] Those skilled in the art related to the embodiments of the present invention will understand that they may be implemented in modified forms without departing from the essential characteristics of the description. Therefore, the disclosed methods should be considered in an illustrative rather than a restrictive sense. The scope of the invention is defined by the claims, not by the detailed description of the invention, and all variations within the scope of the claims should be interpreted as being included within the scope of the invention. Explanation of the symbols
[0124] 1: Policy generator for SLM 100: Central server 110: Processor 111: Base data generation unit 112: Policy Learning Department 113: Policy Creation Department 114: Memory 120: Communication unit 130: Input / Output Interface 200: Terminal 210: Mobile terminal 220: Computing terminal 230: Workstation 240: Agent Server
Claims
Claim 1 A method for generating a policy for an SLM comprising: a step of receiving a professional dataset; a step of generating a basis dataset based on the professional dataset and a pre-stored initial basis set; a step of verifying the basis dataset through a self-validation function; a step of learning an inference policy through an embodied knowledge graph based on the verified basis dataset; a step of learning a planning policy based on the basis set of the learned inference policy and the planning policy reconstruction loss; and a step of generating an SLM policy including a final inference policy and a planning policy based on the learned inference policy and planning policy; wherein, in the step of learning the planning policy, the basis set of the inference policy is generated based on an encoder policy based on an encoder prompt pool, a decoder policy based on a decoder prompt pool, and an attention module while performing the inference policy learning step. Claim 2 delete Claim 3 A method for generating a policy for an SLM according to claim 1, wherein the encoder prompt pool includes a prefix prompt and a postfix prompt. Claim 4 A policy generation method for SLM according to claim 1, characterized in that the set of grounds for the inferred inference policy is generated through grounds reconstruction loss based on a graph extracted through grounds and a knowledge graph search function. Claim 5 In Paragraph 4, the above-mentioned basis reconstruction loss is a mathematical formula (Here, :Reconstruction loss, o: Observation, h: Task description, R: Set of grounds, A method for generating a policy for an SLM characterized by : basis, g: graph extracted through a knowledge graph search function). Claim 6 A method for generating a policy for an SLM, characterized in that, in the step of learning the inference policy according to claim 1, the embodied knowledge graph is a prompted knowledge graph. Claim 7 A method for generating a policy for an SLM according to claim 6, wherein the prompted knowledge graph is based on a batch sample comprising positive pairs, which are embodied knowledge graphs executing the same plan, and negative pairs, which are successive plan steps. Claim 8 In claim 6, the aforementioned prompted knowledge graph is characterized by being based on contrastive learning loss, and the aforementioned contrastive learning loss is a mathematical formula (Here, :Contrast learning loss, : Batch sample, : embedding space, d: embedding space The sum of distance metrics corresponding to the elements of each ground embedding sequence within ∈Z, A method for generating a policy for an SLM characterized by being a : margin parameter. Claim 9 In paragraph 1, in the step of learning the planning policy, the planning policy is learned by predicting the next plan (a) based on the basis set of the learned inference policy, and a mathematical formula (Here, :Planned policy, R: Basis set of inferred policy, A policy generation method for SLM characterized by being learned through : inference policy, g: graph extracted through a knowledge graph search function, a: next plan). Claim 10 In Paragraph 1, the above-mentioned planning policy reconfiguration loss is a mathematical formula (Here, : Planning policy restructuring loss, o: observation, h: task description, R: basis set, A method for generating a policy for SLM characterized by :plan policy, a: next plan). Claim 11 A policy generation device for SLM comprising: an input / output interface for receiving a professional dataset; a processor for generating an SLM policy based on the professional dataset; and a communication unit for transmitting the generated SLM policy to a terminal; wherein the processor generates a basis dataset based on the professional dataset and a pre-stored initial basis set, verifies the basis dataset through a self-verification function, learns an inference policy through an embodied knowledge graph based on the verified basis dataset, learns a planning policy based on the basis set of the learned inference policy and the planning policy reconstruction loss, generates an SLM policy including a final inference policy and a planning policy based on the learned inference policy and the planning policy, and when learning the planning policy, the basis set of the inference policy is generated based on an encoder policy based on an encoder prompt pool, a decoder policy based on a decoder prompt pool, and an attention module while performing the inference policy learning step. Claim 12 A policy generation device for an SLM, characterized in that, in claim 11, the encoder prompt pool includes a prefix prompt and a postfix prompt. Claim 13 A policy generation device for an SLM, wherein, in claim 11, the processor is characterized in that the basis set of the inferred policy is generated through a basis reconstruction loss based on a graph extracted through a basis and knowledge graph search function. Claim 14 In Paragraph 13, the above processor, the above basis reconstruction loss is a mathematical formula (Here, :Reconstruction loss, o: Observation, h: Task description, R: Set of grounds, A policy generation device for SLM characterized by : basis, g: graph extracted through a knowledge graph search function. Claim 15 A policy generation device for an SLM, characterized in that, in claim 11, when the processor learns the inference policy, the embodied knowledge graph is a prompted knowledge graph. Claim 16 A policy generation device for an SLM, wherein, in paragraph 15, the processor is characterized in that the prompted knowledge graph is based on a batch sample comprising positive pairs, which are embodied knowledge graphs executing the same plan, and negative pairs, which are successive plan steps. Claim 17 In paragraph 15, the processor is characterized in that the prompted knowledge graph is based on contrastive learning loss, and the contrastive learning loss is a mathematical formula (Here, :Contrast learning loss, : Batch sample, : embedding space, d: embedding space The sum of distance metrics corresponding to the elements of each ground embedding sequence within ∈Z, A policy generation device for an SLM characterized by being : margin parameter. Claim 18 In paragraph 11, when the processor learns the planning policy, the planning policy is learned by predicting the next plan (a) based on the basis set of the learned inference policy, and a mathematical formula (Here, :Planned policy, R: Basis set of inferred policy, A policy generator for an SLM characterized by being learned through : inference policy, g: graph extracted through a knowledge graph search function, a: next plan). Claim 19 In Clause 11, the above processor, the above planning policy reconfiguration loss is a mathematical formula (Here, : Planning policy restructuring loss, o: observation, h: task description, R: basis set, A policy generation device for SLM characterized by :plan policy, a: next plan). Claim 20 A central server for generating a policy for an SLM, comprising: an input / output interface for receiving a professional dataset; a processor for generating an SLM policy based on the professional dataset; and a communication unit for transmitting the generated SLM policy to a terminal; wherein the processor generates a basis dataset based on the professional dataset and a pre-stored initial basis set, verifies the basis dataset through a self-verification function, learns an inference policy through an embodied knowledge graph based on the verified basis dataset, learns a planning policy based on the basis set of the learned inference policy and the planning policy reconstruction loss, generates an SLM policy including a final inference policy and a planning policy based on the learned inference policy and the planning policy, and when learning the planning policy, the basis set of the inference policy is generated based on an encoder policy based on an encoder prompt pool, a decoder policy based on a decoder prompt pool, and an attention module while performing the inference policy learning step.
