Action space alignment method and system of body agent based on large language model

By combining internal alignment and external alignment methods, efficient parameter fine-tuning and retrieval generation technology are used to solve the illusion problem in the alignment of the action space of the embodied agent, the efficiency and interpretability of the model are improved, and the effectiveness of the agent's behavior is ensured. and security.

CN119005359AActive Publication Date: 2024-11-22YUNNAN UNITED VISION TECH CO LTD

Patent Information

Application Number
CN202411116365.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-14
Publication Date
2024-11-22
Estimated Expiration
2044-08-14

AI Technical Summary

Technical Problem

Embodied agents based on large language models have misleading 'illusion' behavior in action space alignment, and existing alignment methods such as supervised fine-tuning and reinforcement learning face challenges such as high data preparation burden, high computing resource consumption, and sparse reward problems. It is difficult to ensure that agent behavior is consistent with human expectations.

Method used

The collaborative method of internal alignment and external alignment is adopted to ensure the effectiveness and security of the actions generated by the agent through efficient parameter fine-tuning and retrieval generation methods. Internal alignment is efficiently fine-tuned through Q-Lora parameters to learn expert demonstration trajectory, external alignment is selected through ROUGE-L scores and In-Context Learning to reduce hallucination phenomena.

Benefits of technology

It improves the flexibility and resource efficiency of the model, reduces the need for designing reward functions and reinforcement learning training, reduces the consumption of computing resources, enhances the interpretability and effectiveness of the model, and effectively alleviates the problem of action hallucinations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119005359A_ABST
    Figure CN119005359A_ABST
Patent Text Reader

Abstract

The invention relates to an action space alignment method and system for an agent with a body based on a large language model, and belongs to the technical field of reasoning and planning. The method comprises internal alignment and external alignment; in the internal alignment, efficient model self-adaption of the intelligent agent is promoted through an efficient parameter fine tuning method; in the external alignment, the action output by the intelligent agent is ensured to be aligned with a safe and effective action set through a method based on retrieval generation. According to the method, the effectiveness of the action generated by the intelligent agent is ensured, the overall performance of the model is enhanced, and the effectiveness is proved; in addition, the method needs lower computing resources; the controllability and the interpretability of model output are also improved; the internal and external alignment method provides an effective solution for relieving motion illusion with a body based on llm.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to an action space alignment method of an embodied intelligent agent based on a large language model, and belongs to the technical field of reasoning and planning. Background Art

[0002] In recent years, representative large language models (LLMs) such as GPT-4, Claude-3, Gemini, and LlaMA have demonstrated excellent performance in a wide range of natural language processing and generation tasks. They not only perform well in natural language processing, but also develop rapidly in areas such as reasoning and planning, bringing new vitality to embodied intelligence technology. Many recent studies have explored the capabilities of embodied agents based on LLMs in intent understanding, logical reasoning, and task planning, demonstrating the potential of large language models as core planners in embodied agents.

[0003] However, aligning the capabilities and behaviors of embodied agents with safe and effective action spaces remains a major challenge. Studies have shown that these model-based agents may inadvertently learn biased, discriminatory, or harmful content from the training data, causing outputs to deviate from human expectations. In particular, these embodied agents may produce misleading "hallucination" behaviors and create erroneous or unfounded content. Specifically, the action "hallucinations" of the agent mainly include format hallucinations, that is, an incorrect control action; and state hallucinations, that is, unexecutable actions based on incorrect state judgments. For example Figure 1 As shown in the figure, an unaligned agent may generate an incorrectly formatted action instruction "gotocoffeetable 1", while the correct instruction format should contain a space between "go" and "to". This format error prevents the agent from successfully reaching "coffeetable 1" and further triggers state hallucination, resulting in an unexecutable action instruction "put vase 3in / on coffeetable 1". Because the agent is not actually at the location of "coffeetable 1", it cannot complete the action. Therefore, it is particularly important to use effective alignment techniques to ensure that the behavior of agents based on large language models is consistent with a feasible and safe operation space.

[0004] The widely used model alignment methods are mainly based on the following alignment paradigm: supervised fine-tuning (SFT) first, followed by reinforcement learning (RL). This method first fine-tunes the model on training data in a specific domain, aiming to improve its practicality by enhancing its ability to follow instructions and to alleviate hallucinations to a certain extent. Nevertheless, when faced with unfamiliar inputs, models that rely solely on supervised fine-tuning may still give wrong answers.

[0005] There are several major challenges in using reinforcement learning (RL) to achieve action space alignment of embodied agents based on large language models in embodied settings:

[0006] (1) Customized reward functions are required: RL usually requires customized reward functions for different environments, which may require collecting additional datasets or performing complex manual annotations. This step not only increases the burden of preparation, but may also introduce human bias.

[0007] (2) High consumption of computing resources: RL training usually requires a lot of computing resources. Since the size of a general reward model may be comparable to the language model itself, reinforcement learning training is particularly expensive in terms of resource consumption, and the training process often lacks stability, which further increases the difficulty and cost of training.

[0008] (3) There is a sparse reward problem: In an embodied environment, the agent may encounter a sparse reward problem, that is, it is difficult for the correct action or behavior sequence to be rewarded through direct feedback. This situation is not conducive to RL training because it leads to inefficient learning and it is difficult for the agent to find an effective learning path in a complex environment;

[0009] In order to more effectively ensure that the model output is consistent with human preferences, many studies have added a reinforcement learning stage based on human feedback after supervised fine-tuning. For example, InstructGPT, RAFT, and Constitutional AI use human feedback for reinforcement learning to train models to learn and understand human preferences. This method improves the performance and output quality of the model by training a reward model to guide the learning process based on human evaluation of the model output. Summary of the invention

[0010] In order to solve the above problems, the present invention provides an action space alignment method and system for embodied intelligent agents based on a large language model, which uses internal alignment and external alignment in coordination, which not only ensures the effectiveness of the action generated by the intelligent agent, but also enhances the overall performance of the model. The internal and external alignment method is used to alleviate the action illusion problem of embodied intelligent agents based on LLM.

[0011] The technical solution of the present invention is: in a first aspect, the present invention provides an action space alignment method of an embodied intelligent agent based on a large language model, including internal alignment and external alignment;

[0012] In the internal alignment, an efficient parameter fine-tuning method is used to promote efficient model adaptation of the agent;

[0013] In the external alignment, a retrieval-based generation method is used to ensure that the actions output by the agent are aligned with the safe and effective action set.

[0014] Furthermore, in the internal alignment, the efficient fine-tuning method of parameters is used to promote efficient model adaptation of the agent, specifically including:

[0015] The Q-Lora parameter efficient fine-tuning method PEFT is used to promote efficient model adaptation of the intelligent agent;

[0016] The goal of internal alignment is to train an LLM-based embodied agent π on the embodied dataset θ , which learns from expert demonstrations in solving embodied reasoning tasks x task The expert demonstration trajectory segments are presented in the form of action-environment state pairs, where each pair of data represents the t Then the corresponding environmental state s t ; Assume that τ t =[(a0,s0),(a1,s1),(a2,s2),...,(a t-1 ,s t-1 )] is represented as a trajectory segment at time t; the goal of supervised fine-tuning is to learn to generate an action sequence {a t ~π θ (x task ,τ t )} to complete the task efficiently; in short, this is to seek to find a set of parameters θ * , so that in the embodied agent π θ Guided loss function minimize;

[0017]

[0018] Select cross entropy loss as the loss function It measures π θ and the difference between the actions produced by the expert demonstration, In the embodied agent π θ The expected value or expected loss under the strategy, Θ represents the parameter space, which contains all possible parameter sets θ.

[0019] Furthermore, in the external alignment, the method based on retrieval generation is used to ensure that the actions output by the agent are aligned with the safe and effective action set, specifically including:

[0020] In the retrieval-generation-based method, the goal of retrieval-enhancement generation is to select the best action from a set of safe and effective actions as the final output of the model; when the internally aligned agent π θ Generate a candidate action a c When , the first step of external action space alignment is to calculate the candidate action a cThe ROUGE-L score between each valid action v in the valid action space V, where the ROUGE-L score is used to evaluate the similarity between two sentences, reflecting a c The similarity between all valid actions v; the calculation formula of ROUGE-L score is as follows:

[0021]

[0022] Here, LCS(a c ,v) represents the candidate action a c and the length of the longest common subsequence between valid actions v, len(a c ) is a candidate action a c The length of a c and the ROUGE-L score between valid actions v, the action with the highest score is selected as the final output of the model; if multiple valid actions are c When they have the highest and equal similarity values, a selected action set C is formed.

[0023] Furthermore, after obtaining the selected action set C, the strategy model π is introduced p To select the best action; use the In-Context Learning (ICL) capability of large language models to align action space; the introduction of ICL aims to enhance the performance of the model on new tasks by modifying the input query of the model instead of directly updating the model weights; the currently retrieved valid action set C is used as external input to allow LLM to obtain information about the physical world outside the execution context, thereby anchoring LLM's response to the retrieved action and reducing the possibility of hallucination phenomena; the final action is represented as:

[0024] a f =π p (x task ,τ t ,C)

[0025] Among them, x task represents the embodied reasoning task, τ t Represented as a trajectory segment at time t.

[0026] In a second aspect, the present invention provides an action space alignment system for an embodied intelligent agent based on a large language model, comprising a module for executing the method described in the first aspect above.

[0027] In a third aspect, the present invention provides a processor for running a program, wherein the program, when running, executes the action space alignment method of an embodied intelligent agent based on a large language model as described in any one of the first aspects.

[0028] In a fourth aspect, the present invention provides a computer-readable storage medium, comprising a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the action space alignment method of an embodied intelligent agent based on a large language model as described in the first aspect.

[0029] The beneficial effects of the present invention are:

[0030] 1. Compared with the traditional alignment method based on reinforcement learning (RL), the newly proposed internal and external alignment strategy shows higher flexibility and resource efficiency, and contributes to improving the interpretability of the model. The core advantages are as follows:

[0031] First, it eliminates the need to design reward functions and perform reinforcement learning (RL) training on specific specialized tasks.

[0032] Second, the choice of policy models within the framework is flexible. It can be either a fine-tuned open source small model (model parameter is about 7B) or a powerful commercial large language model.

[0033] Third, retrieval based on effective action space can generate actions by retrieving evidence from the physical environment (such as a safe and effective action set), which naturally reduces the risk of generating hallucinatory content. Finally, the strategy based on retrieval generation can determine the source of the LLM-generated answers and enhance the interpretability and transparency of the model's decision-making process.

[0034] 2. The present invention not only ensures the effectiveness of the agent-generated actions, but also enhances the overall performance of the model, proving its effectiveness; in addition, the method of the present invention requires lower computing resources: during internal alignment, an efficient parameter fine-tuning method is utilized; while in external alignment, compared with the traditional reinforcement learning-based model alignment technique, the retrieval-generated external alignment shows higher efficiency and stability; in addition, it also improves the controllability and interpretability of the model output; the internal and external alignment method provides an effective solution to alleviate the LLM-based embodied body action illusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is the internal and external alignment framework diagram in the present invention;

[0036] Figure 2 This is an example of embodied agent action space alignment based on a large language model in the present invention. DETAILED DESCRIPTION

[0037] Example 1: Figure 1-2 As shown, in the first aspect, the present invention provides an action space alignment method of an embodied intelligent agent based on a large language model, including internal alignment and external alignment;

[0038] In the internal alignment, an efficient parameter fine-tuning method is used to promote efficient model adaptation of the agent;

[0039] The efficient parameter fine-tuning method is used to promote efficient model adaptation of intelligent agents, including: Figure 2 Specifically, in the internal alignment, in order to effectively balance the computational cost and prevent the catastrophic forgetting problem in the base model, the Q-Lora parameter efficient fine-tuning method PEFT is used to promote efficient model adaptation of the agent;

[0040] The goal of internal alignment is to train an LLM-based embodied agent π on the embodied dataset θ , which learns from expert demonstrations in solving embodied reasoning tasks x task The expert demonstration trajectory fragments are presented in the form of action-environment state pairs, where each pair of data represents the t Then the corresponding environmental state s t ; Assume that τ t =[(a0,s0),(a1,s1),(a2,s2),...,(a t-1 ,s t-1 )] is represented as a trajectory segment at time t; the goal of supervised fine-tuning is to learn to generate an action sequence {a t ~π θ (x task ,τ t )} to complete the task efficiently; in short, this is to seek to find a set of parameters θ * , so that in the embodied agent π θ Guided loss function minimize;

[0041]

[0042] Select cross entropy loss as the loss function It measures π θ and the difference between the actions produced by the expert demonstration, In the embodied agent π θ The expected value or expected loss under the policy, Θ represents the parameter space, which contains all possible parameter sets θ. This internal alignment method not only reduces the action illusion of the agent, but also allows the embodied agent to make reasonable actions in the new environment.

[0043] In the external alignment, this is different from the traditional method of updating model parameters using reinforcement learning. At this stage, a retrieval-based generation method is used to ensure that the actions output by the agent are aligned with the safe and effective action set. Specifically, it includes:

[0044] In the retrieval-generation-based method, the goal of retrieval-enhancement generation is to select the best action from a set of safe and effective actions as the final output of the model; when the internally aligned agent π θ Generate a candidate action z c When , the first step of external action space alignment is to calculate the candidate action z c The ROUGE-L score between each valid action v in the valid action space V, where the ROUGE-L score is used to evaluate the similarity between two sentences, reflecting z c The similarity between all valid actions v; the calculation formula of ROUGE-L score is as follows:

[0045]

[0046] Here, LCS(z c ,v) represents the candidate action z c and the length of the longest common subsequence between valid actions v, len(a c ) is a candidate action a c The length of a c and the ROUGE-L score between valid actions v, the action with the highest score is selected as the final output of the model; if multiple valid actions are c When they have the highest and equal similarity values, a selected action set C is formed.

[0047] Furthermore, after obtaining the selected action set C, the strategy model π is introduced p To select the best action; use the In-Context Learning, or ICL, capability of a large language model to align the action space; the introduction of ICL aims to enhance the performance of the model on new tasks by modifying the input query of the model, rather than directly updating the model weights; compared with simple ICL prompt engineering, the results generated by retrieval-based ICL are more accurate and the possibility of model hallucination is lower. This is because taking the currently retrieved valid action set C as external input allows LLM to obtain information about the physical world outside the execution context, thereby anchoring LLM's response to the retrieved action and reducing the possibility of hallucination; the final action is represented as:

[0048] a f =π p (x task ,τ t ,C)

[0049] Among them, x task represents the embodied reasoning task, τ tRepresented as a trajectory fragment at time t. In this way, the model can achieve accurate and practical action alignment in the field of embodied intelligence while maintaining high-quality output and ensuring the safety and effectiveness of the action.

[0050] In a second aspect, the present invention provides an action space alignment system for an embodied intelligent agent based on a large language model, comprising a module for executing the method described in the first aspect above.

[0051] In a third aspect, the present invention provides a processor for running a program, wherein the program, when running, executes the action space alignment method of an embodied intelligent agent based on a large language model as described in any one of the first aspects.

[0052] In a fourth aspect, the present invention provides a computer-readable storage medium, comprising a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the action space alignment method of an embodied intelligent agent based on a large language model as described in the first aspect.

[0053] Figure 2 This is an example of embodied agent action space alignment based on a large language model in the present invention. In the figure, the left side is an unaligned agent, and the right side is an aligned agent. Two typical action illusions: (a) format illusion, the agent generates an action with an incorrect format, such as the missing space symbol in "goto coffeetable 1" (b) state illusion, when the agent misjudges its state and attempts an unexecutable action, such as placing an object where it has not yet reached. Aligned agents can alleviate the above two illusions. First, for format illusion, the aligned agent will find the action with the highest similarity in the valid action set to replace it. All actions in the valid action set must be correct / valid in format. Second, for state illusion, if the agent does not successfully reach the table, there cannot be any table-related operations in the valid action set. The aligned agent (based on external alignment generated by retrieval enhancement) ensures that the final action output will only come from the valid action set, so the present invention solves the state environment problem in principle.

[0054] In order to verify the effectiveness of the present method, the present invention has conducted extensive experiments on the #Unseen and #Seen test tasks in the ALFWorld simulation environment. The tasks in #Seen are very similar to the rooms in which the agent was trained, but with different object locations, quantities, etc., while the #Unseen tasks involve performing new tasks in completely unseen rooms and objects. In this environment, the agent needs to generate a series of actions based on instructions and text / visual image observations of each step to effectively complete the task. The agent has a maximum of 30 executable steps to complete the task. The present invention mainly evaluates two types of agents. The first is the basic model, that is, the model that has only undergone efficient parameter fine-tuning (first-stage alignment), and the second is the aligned model, that is, the model has undergone two stages of alignment. Table 1 gives the completed experimental results. )

[0055] Table 1

[0056]

[0057] As shown in Table 1, item 1, the internally aligned model (i.e., the SFT model) is endowed with the basic ability to complete embodied tasks. For example, OPT and Bloomz achieve SR performance of 79.10% and 48.51% in unknown scenarios, respectively. However, relying solely on parameter-efficient fine-tuning, the model may produce invalid "hallucination" actions, with the degree of relief reflected by the LC score. In particular, LlaMA-2 performs poorly in LC, not exceeding 50% in both seen and unseen tasks. This highlights the limitations of relying solely on internal alignment of the model to ensure safe and effective action generation. Therefore, it is necessary to address these challenges through external alignment.

[0058] After RAG-based external alignment, as shown in item 2 of Table 1, the ability of all models to output valid actions is significantly improved, while most indicators experience different degrees of enhancement. By adopting RAG-based external alignment, we ensure that the actions produced by the LLM-based embodied agent are safe and effective, as evidenced by the LC reaching 100% for all models. In addition, with the improvement of the ability to output valid actions, the SR, GCS, and RDI indicators of all models are significantly improved. For example, in the unseen scenario, with the improvement of LC, the SR of LlaMA increases sharply from 38.81% to 75.37%, the GCS increases from 47.51% to 81.34%, and the RDI decreases from 9.70% to 2.20%.

[0059] In summary, extensive experiments on multiple models have verified that our method not only ensures the effectiveness of agent-generated actions, but also enhances the overall performance of the model, proving its effectiveness. In addition, our method requires lower computational resources: During internal alignment, we utilize an efficient parameter fine-tuning method. While in external alignment, retrieval-generated external alignment shows higher efficiency and stability compared to traditional reinforcement learning-based model alignment techniques. In addition, it also improves the controllability and interpretability of model output. The internal and external alignment method provides an effective solution to alleviate the action hallucinations of embodied bodies based on LLM.

[0060] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. An action space alignment method for embodied agents based on a large language model, characterized by: Including internal alignment and external alignment; In the internal alignment, an efficient parameter fine-tuning method is used to promote efficient model adaptation of the agent; In the external alignment, a retrieval-based generation method is used to ensure that the actions output by the agent are aligned with the safe and effective action set.

2. The method for aligning action spaces of embodied agents based on a large language model according to claim 1, characterized in that: In the internal alignment, the efficient fine-tuning method of parameters is used to promote efficient model adaptation of the agent, specifically including: The Q-Lora parameter efficient fine-tuning method PEFT is used to promote efficient model adaptation of the intelligent agent; The goal of internal alignment is to train an LLM-based embodied agent π on the embodied dataset θ , which learns from expert demonstrations in solving embodied reasoning tasks x task The expert demonstration trajectory segments are presented in the form of action-environment state pairs, where each pair of data represents the t Then the corresponding environmental state s t ; Assume that τ t =[(a0,s0),(a1,s1),(a2,s2),...,(a t-1 ,s t-1 )] is represented as a trajectory segment at time t; the goal of supervised fine-tuning is to learn to generate an action sequence {a t ~π θ (x task ,τ t )} to complete the task efficiently; in short, this is to seek to find a set of parameters θ * , so that in the embodied agent π θ Guided loss function minimize; Select cross entropy loss as the loss function It measures π θ and the difference between the actions produced by the expert demonstration, In the embodied agent π θ The expected value or expected loss under the strategy, Θ represents the parameter space, which contains all possible parameter sets θ.

3. The method for aligning action spaces of embodied agents based on a large language model according to claim 1, characterized in that: In the external alignment, the method based on retrieval generation is used to ensure that the actions output by the agent are aligned with the safe and effective action set, specifically including: In the retrieval-generation-based method, the goal of retrieval-enhancement generation is to select the best action from a set of safe and effective actions as the final output of the model; when the internally aligned agent π θ Generate a candidate action z c When , the first step of external action space alignment is to calculate the candidate action a c The ROUGE-L score between each valid action v in the valid action space V, where the ROUGE-L score is used to evaluate the similarity between two sentences, reflecting z c The similarity between all valid actions v; the calculation formula of ROUGE-L score is as follows: Here, LCS(a c ,v) represents the candidate action a c and the length of the longest common subsequence between valid actions v, len(a c ) is a candidate action a c The length of a c and the ROUGE-L score between valid actions v, the action with the highest score is selected as the final output of the model; if multiple valid actions are c When they have the highest and equal similarity values, a selected action set C is formed.

4. The method for aligning action spaces of embodied agents based on a large language model according to any one of claims 1 or 3, characterized in that: After obtaining the selected action set C, the strategy model π is introduced p To select the best action; use the In-Context Learning (ICL) capability of large language models to align action space; the introduction of ICL aims to enhance the performance of the model on new tasks by modifying the input query of the model instead of directly updating the model weights; the currently retrieved valid action set C is used as external input to allow LLM to obtain information about the physical world outside the execution context, thereby anchoring LLM's response to the retrieved action and reducing the possibility of hallucination phenomena; the final action is represented as: a f =π p (x task ,t t ,C) Among them, x task represents the embodied reasoning task, τ t Represented as a trajectory segment at time t.

5. An action space alignment system for embodied agents based on a large language model, characterized by: The method comprises a module for executing the method according to any one of claims 1 to 4.

6. A processor, characterized in that: The processor is used to run a program, wherein the program, when running, executes the action space alignment method of an embodied intelligent agent based on a large language model as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the action space alignment method of an embodied intelligent agent based on a large language model as described in any one of claims 1 to 4.

Citation Information

Patent Citations

  • Robot optimization control method based on reinforcement learning and evolutionary algorithm

    CN114879486A

  • Reinforcement learning strategy enhancement method based on memory retrieval

    CN117035050A

  • Multi-body agent collaborative game decision-making method based on large language model

    CN118014080A

  • Multi-modal large model-based intelligent task executor training method and system

    CN118350435A

  • Pretraining Already-Pretrained Models for Diverse Downstream Tasks

    US20240256964A1

Cited By

  • Intelligent low-rank adaptive fine-tuning model method

    CN120031126A

  • Intelligent task planning method and device based on environment perception, equipment and medium

    CN120373694A

  • Intelligent agent behavior alignment detection method and system based on thinking chain auditing

    CN121094123A

  • A method and system for detecting alignment of agent behavior based on thought chain auditing

    CN121094123B

  • Large language model (LLM) entity and relationship extraction for fine-tuning llms

    WO2026212979A1