A large language model framework for highlighting playback strategy optimization for artificial intelligence metering scenarios

Abstract

The application discloses a large language model framework for highlighting playback strategy optimization of an artificial intelligence metering scene, relates to the technical field of artificial intelligence, adopts an "Actor-Critic" architecture, shares a frozen large language model, and integrates a low-rank adapter LoRA; action prompts and observation prompts are obtained from highlighting historical strategy data, and corresponding scalar rewards are obtained; input is provided to the large language model, the adapted large language model is fine-tuned by the LoRA; a dynamic normalization module is set, normalization weights are dynamically adjusted, the generation probability of actions is adjusted, and the probability of action prompts related to a task is enhanced. The framework efficiently reuses historical experience, improves the relevance of generated actions to task goals, effectively improves the decision-making ability of an agent, reduces the dependence on real-time data, shortens the model adaptation cycle of a new scene, reduces unnecessary environment interaction, and makes it possible for the large language model to be applied in a resource-limited scene.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to large language model (LLMs) optimization, reinforcement learning and decision task adaptation techniques, and particularly to a large language model framework for optimizing highlight playback strategies for AI econometric scenarios. Background Technology

[0002] In recent years, large language models (LLMs) have achieved groundbreaking results in the field of natural language generation and understanding. Recent research shows that LLMs can not only coordinate other AI models and tools to handle complex multimodal tasks, but also be deployed on robots to enable interactive operations with the real world. While LLMs can provide insightful suggestions for complex tasks, they are limited by alignment issues, which can lead to ineffective generated actions and difficulty in accurately capturing dynamic changes in the environment. Especially when specific constraints exist in the environment, LLMs often fail in decision-making tasks.

[0003] Reinforcement learning, through trial and error in the environment, allows agents to learn policies from scratch, thus ensuring that reinforcement learning agents can effectively align with the environment. However, most traditional reinforcement learning methods start with random policies and rely on environmental feedback for updates. Such policies are typically inefficient in the initial stages.

[0004] In existing technologies, some methods attempt to combine large language models with reinforcement learning, but they have obvious drawbacks: they rely solely on current policy data for updates, failing to fully utilize historical high-quality policy experience, resulting in low sample utilization; in high-dimensional action spaces or dynamic environments, policy updates are prone to exceeding constraints, leading to performance fluctuations.

[0005] Therefore, there is an urgent need for an optimization framework that can integrate the reuse of historical experience with the characteristics of large language models to solve the above-mentioned alignment and sample efficiency problems, and improve the stability of policies and learning efficiency in complex decision-making tasks.

[0006] In this disclosure, to validate the constructed framework, four challenging multidimensional NLP benchmark sets were selected, including ARC_C, HellaSwag, PIQA, and MMLU. Among them: The AI2 Reasoning Challenge (ARC) dataset is a multiple-choice question dataset primarily designed to test a model's ability in scientific reasoning. The ARC dataset includes science exam questions from grades 3 to 9, divided into two parts: the easy part (ARC-E) and the challenging part (ARC-C). These two parts evaluate the model's performance when handling easily answerable common-sense questions and when facing more complex scientific reasoning questions, respectively. The ARC-C part, in particular, contains many questions requiring higher levels of reasoning and logical deduction, thus placing significant demands on the language model's reasoning capabilities. Furthermore, the ARC dataset includes 14.3 million KB of unstructured text passages containing substantial background knowledge, further increasing the challenge of the task.

[0007] HellaSwag is a highly challenging dataset for evaluating commonsense natural language reasoning (NLI) capabilities. The tasks on this dataset are typically very simple for humans, achieving over 95% accuracy, but are extremely challenging for large language models. HellaSwag provides a scenario description with several possible outcomes, then asks the model to choose the most appropriate one. This type of task not only examines the model's understanding of common sense but also requires it to make reasonable inferences based on different contexts. Therefore, HellaSwag tests a model's ability to handle complex commonsense reasoning tasks.

[0008] PIQA (Physical Interaction QA) is a physics-based question-answering dataset designed to test a model's reasoning ability in the physical world. PIQA dataset questions are typically based on everyday physical scenarios, requiring the model to predict interactions and reactions between objects. These questions often involve common physical phenomena such as gravity and friction, testing the model's understanding and reasoning ability regarding real-world physical laws.

[0009] MMLU (Massive Multitask Language Understanding) is a comprehensive benchmark for evaluating the performance of language models in multitasking environments. The MMLU dataset covers 57 subject areas, including science, technology, engineering, mathematics, humanities, and social sciences, and also tests the model's ability to process world knowledge and solve real-world problems. MMLU not only tests the model's performance on zero-shot and few-shot tasks but also examines its ability to handle diversity across multiple disciplines. MMLU aims to evaluate the model's language understanding and reasoning abilities across various domains in a manner similar to human learning. The test covers topics ranging from traditional fields such as mathematics and history to more specialized disciplines such as law and ethics, greatly expanding the testing scope of language models. Summary of the Invention

[0010] This disclosure provides a framework for optimizing highlighting and playback strategies in AI econometrics scenarios. It aims to address the alignment problem of large language models in decision-making tasks within AI econometrics contexts, specifically the issues of generating effective actions, adapting to dynamic environments, and improving sample efficiency in reinforcement learning. By integrating highlighting and playback strategy optimization methods with the language understanding capabilities of large language models, efficient and stable strategy optimization is achieved while preserving the original natural language processing capabilities of the large language model. This framework is particularly suitable for solving alignment problems in complex decision-making tasks, such as virtual environment interaction, multi-step task planning, and home service robot control, namely, generating effective actions and accurately understanding environmental feedback.

[0011] The HiPLM framework based on highlight playback strategy optimization proposed in this disclosure has the following main architecture: During the training phase: the Actor-Critic framework is adopted, sharing the frozen large language model, and integrating the low-rank adapter LoRA to achieve efficient parameter fine-tuning; specifically including: The large language model serves as the foundation of the framework; The Actor module is used to generate action decisions for agents in the environment, and consists of a frozen large language model plus LoRA. The Critic module consists of an additional multilayer perceptron added after the last Transformer block of the large language model; the Critic module is jointly trained with the highlighting history strategy data, and the TV distance is used as the reward constraint. A replay buffer is used to store experiential data generated by the agent through its interaction with the environment; The replay buffer provides highlighted historical policy data, including action cues and observation cues, as well as corresponding scalar rewards, which are then fed into the large language model. LoRA fine-tunes the adapted large language model to improve the agent's decision-making ability. During the inference phase, only the Actor module is retained, while the Critic module is discarded.

[0012] Furthermore, the input to the large language model includes action cues and observation cues, wherein each action cue... A sequence consisting of multiple tokens Composition, in which This indicates the length of the action cue; the length of the action cue can vary; in each step, the observed cue... Possibly related to the currently selected action Related, through effective actions Concatenation is used as input to a large language model; The large language model generates token-level probabilities, which are used to calculate the probability of action cues. The token-level probabilities are as follows:

[0013] Typically, large language models provide scores based on the log-likelihood of each token, i.e., the log probability. ; The action is normalized using the Softmax function to obtain the policy. : .

[0014] Furthermore, the framework also includes a dynamic normalization module, used to dynamically adjust the normalization weights to adjust the generation probability of actions, wherein the probability of action cues related to the task is increased and interference from actions unrelated to the task is reduced.

[0015] Furthermore, in the dynamic normalization module, the formula for calculating the joint probability is as follows:

[0016] in, Action prompts The number of words in Action prompts Dynamic weights, , The correlation between actions and the current task objective:

[0017] In the formula, Representative actions With observation tips The similarity between them; The similarity calculation uses the BERT model, which transforms each action cue and task objective into a high-dimensional vector representation. Then, the cosine similarity between these vectors is calculated to quantify their relevance. The formula for calculating cosine similarity is:

[0018] in, and These represent the embedding vectors for action cues and task objectives, respectively. This represents the dot product operation. The norm of a vector.

[0019] Furthermore, LLaMA-7B is adopted as the basic model of the framework.

[0020] Furthermore, in the Actor module, the function of LoRA is represented as follows: Let LLaMA-7B be the pre-trained model, and the weight matrix be... ; LoRA breaks it down into:

[0021] in, For the newly added low-rank matrix, ; For rank, the total number of parameters starts from Reduce to ; Maintain pre-trained weights No change, only train the two inserted low-rank matrices and ; Input data After the original weights and adapter The combined effect of these factors yields the output vector. : .

[0022] Furthermore, the training process of the framework includes the following steps: The agent generates experiential data through interaction with the environment, and this experiential data is stored in a replay buffer; When the training rounds reach the set threshold, the highlighted historical experience is sampled from the replay buffer and combined with the new data generated by the current strategy for training. In this step, the data used each time is discarded from the replay buffer. The replay buffer provides the target description corresponding to the highlighted historical strategy as an action cue, the description of the current environment observation as an observation cue, and the corresponding scalar reward. Collect action cues, observation cues, and rewards to provide information to the large language model, and then use the rewards returned by the environment to fine-tune the adapted large language model through LoRA, thereby improving the agent's decision-making ability; To ensure training stability, Critic and Actor employ different learning rates: Critic uses a higher learning rate to quickly adapt to new data and provide stable value estimates, while Actor uses a lower learning rate to ensure smooth policy updates.

[0023] Compared with the prior art, the beneficial effects of this disclosure are: (1) Solving the alignment problem: Through the dynamic normalization module, the correlation between the actions generated by the large language model and the task target is significantly improved. In the Overcooked environment, the effective action generation rate is more than 30% higher than that of traditional methods. (2) Improved sample efficiency: The highlighting and playback mechanism makes full use of historical high-quality experience. In the VirtualHome environment, the number of environmental interactions required to complete the task is reduced by 40% compared with the previous method; (3) Enhanced training stability: The TV distance penalty and hierarchical learning rate design reduce policy update fluctuations by 25% and exhibit more robust convergence performance in high-dimensional action spaces; (4) The online fine-tuning method successfully improved the model’s language understanding and reasoning ability in multiple tasks by fine-tuning the model in a dynamic virtual environment. It showed a clear advantage in tasks that require common sense reasoning and physical reasoning. Attached Figure Description

[0024] The above and other objects, features and advantages of this disclosure will become more apparent from the more detailed description of exemplary embodiments of this disclosure taken in conjunction with the accompanying drawings, in which the same reference numerals generally represent the same components.

[0025] Figure 1 A schematic diagram of a large language model framework optimized according to the highlight playback strategy of this disclosure; Figure 2 Schematic diagram of the experimental environment: Overcooked virtual kitchen environment - tomato salad illustration; Figure 3 Schematic diagram of the experimental environment: Overcooked virtual kitchen environment - tomato - lettuce salad; Figure 4 is a schematic diagram of the experimental environment: VirtualHome family scene - food preparation schematic diagram; Figure 5 is a schematic diagram of the experimental environment: VirtualHome family scene - entertainment activities. Detailed Implementation

[0026] Preferred embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0027] This disclosure proposes a framework for optimizing highlight playback strategies in large language models, aiming to improve the sample efficiency and stability of strategy optimization in large language models.

[0028] In one exemplary embodiment, the core architecture includes: dynamic normalization, a model architecture module, and a training and inference process module, with each module working together to optimize the strategy.

[0029] Further details are as follows: 1. Dynamic normalization: To address the issue that actions directly generated by LLMs may be ineffective in real-world environments, the range of generated actions can be partially limited through cue design, i.e., adding action constraints to the cue. However, most existing LLMs cannot strictly adhere to these constraints, especially small and medium-sized LLMs with models smaller than 65 bytes. Therefore, new methods are needed to generate effective policies that meet environmental requirements.

[0030] Cue design refers to designing each action as an action cue, such as "pick up the tomato," and combining it with observation cues, such as "you see a tomato," as input.

[0031] In this embodiment, each action prompt Sequences consisting of multiple tokens Composed of, among which This indicates the length of the action cue, and the length of the action cue can be different. In each step, the observed... Possibly related to the currently selected action Related, through effective actions The input to the large language model was constructed by splicing together the data.

[0032] The token-level probabilities generated by LLMs are used to calculate the probability of an action cue, which differs from the probability of the action actually being executed. Action Cue The token-level probabilities are as follows:

[0033] Typically, LLMs provide scores that are the log-likelihood of each token, i.e., the log probability. The policy is obtained by normalizing the actions using the Softmax function. :

[0034] To address the impact of varying action cue lengths on probability, previous methods introduced Word normalization to balance the probability distribution of action cues of different lengths. While this framework effectively generates action policies through reasonable cue design and normalization strategies, it still has significant limitations when handling high-dimensional action spaces and dynamic environments. Simple Word normalization is a global static strategy that fails to dynamically adjust the importance of action cues based on the task or environment. For example, in a salad-making task, "picking up the tomato" should be more important than "putting down the vegetables," but static normalization cannot reflect this difference. Action generation lacks specificity, easily leading to inefficient policy generation. In dynamic environments, the generation probability of important actions may be underestimated, requiring querying scores for all possible actions one by one. With a large action space, computational costs increase exponentially, resulting in high computational complexity, low sample efficiency, and limited applicability to real-time decision-making scenarios.

[0035] To address the aforementioned issues, this embodiment proposes an improved method: by dynamically adjusting the normalized weights, prioritizing the enhancement of the probability of action prompts related to the task, reducing interference from actions unrelated to the task, and dynamically allocating the weights.

[0036] For example, in a salad-making task, based on the observational cue "You see a tomato," the weight of "picking up the tomato" will be higher than that of "cutting lettuce." By incorporating a dynamic weight term when calculating the joint probability, the generation probability of actions is adjusted, highlighting key action cues and enhancing the strategy's relevance. This allows the strategy to adapt to real-time environmental changes in dynamic tasks.

[0037] After introducing dynamic weights, the logarithmic probability of generating a language token corresponding to a single action (such as the text token corresponding to "pick up the tomato") is obtained by taking the logarithm, as shown in the following formula:

[0038] in, It is an action prompt The number of words in It is an action prompt Dynamic weights, , It is the correlation between the action and the current task objective.

[0039]

[0040] Represents action With observation tips The similarity between them.

[0041] In this embodiment, the pre-trained language model BERT is used for modeling. As a Transformer-based pre-trained language model, BERT can efficiently capture contextual information, handle long-distance semantic relationships, and learn rich linguistic knowledge through a pre-training corpus. BERT has already been trained on large-scale text data, thus enabling it to generate embedding vectors with deep semantic information for input task cues and action cues in an unsupervised manner. These embedding vectors serve as the basis for task relevance evaluation, accurately reflecting the semantic matching degree between the task objective and action cues, ensuring that task-relevant action cues receive higher scores, thereby more effectively guiding the model to perform the task.

[0042] Specifically, the BERT model is used to transform each action cue and task objective into a high-dimensional vector representation, and then their relevance is quantified by calculating the cosine similarity between these vectors. Cosine similarity measures the directional similarity between two vectors, and its calculation formula is as follows:

[0043] in, and These are the embedding vectors for action cues and task objectives, respectively. This represents the dot product operation. This represents the norm of the vector. This method ensures that task-related action cues are given higher weights, enabling the model to better perform its task objectives.

[0044] 2. Model Architecture Module: In practical applications, frequent policy updates can lead to instability during training, especially in high-dimensional and complex environments where policy changes may exceed theoretically set constraints. This instability is particularly pronounced when training large models, as the numerous parameters of large language models can further amplify the problem. Furthermore, in reinforcement learning tasks, sample acquisition is typically expensive, and PPO (Programmable Optimization Process) only utilizes recent interaction data for policy updates, failing to fully leverage historical policy data and effectively extract potential information from high-quality historical policies, resulting in low sample efficiency.

[0045] To address these issues, this embodiment proposes applying the Highlight Playback (HiPPO) method to LLaMA-7B, constructing a large language model framework (HiPLM framework) for highlight playback strategy optimization. The core structure employs an "Actor-Critic" framework, sharing the frozen large language model (such as LLaMA-7B), and integrating a low-rank adapter (LoRA) to achieve efficient parameter fine-tuning.

[0046] In this embodiment, LLaMA-7B was chosen as the base model for the HiPLM framework, primarily based on its advantages in open source, architectural efficiency, and deep compatibility with reinforcement learning. Compared to closed-source models, LLaMA's open source nature allows for flexible adjustments to the model structure and fine-tuning strategies. LLaMA employs a decoder-only Transformer architecture, whose autoregressive generative characteristics are highly compatible with the sequence decision-making requirements of reinforcement learning.

[0047] HiPPO significantly improves the sample efficiency and training stability of PPO by introducing a highlight replay mechanism and reward constraint optimization. Specifically: The highlight replay mechanism fully leverages high-quality experience data from historical strategies. By constructing a replay buffer, it dynamically stores and prioritizes the use of high-quality historical strategies, thereby reducing reliance on real-time environment interaction and significantly improving sample efficiency. Furthermore, the reward-constrained optimization mechanism can be well integrated with LLaMA. By introducing the Total Variation distance (TV distance) as a penalty term during the optimization process, the magnitude of policy updates is effectively constrained. This mechanism can smooth the policy update process and alleviate instability issues during training, especially in the LoRA low-rank parameter fine-tuning of LLaMA, avoiding performance degradation caused by excessive policy perturbations.

[0048] In the HiPLM architecture, both the actors and critics of this system incorporate the core mechanisms of HiPPO to more efficiently utilize historical policy data and enhance training stability. The specific design is as follows: Figure 1 As shown in the diagram, the architecture of the shared frozen large language model by actors and critics is illustrated, including the integration method of the LoRA low-rank adapter and the data flow of the highlighting playback mechanism.

[0049] Specifically: In this embodiment, the critic module is similar to the original design, with an additional multilayer perceptron (MLP) added after the last Transformer block of the LLaMA-7B model. The critic is jointly trained with the highlighted policy data, and TV distance is introduced as a reward constraint to reduce fluctuations in the policy optimization process and ensure that the critic's estimation of the value function is smooth and stable.

[0050] The Actor module consists of a frozen LLaMA-7B model plus LoRA parameters. LoRA is a parameter- and computationally efficient fine-tuning method that incorporates trainable rank factorization matrices into each layer of a large language model. By freezing the original model parameters, only a small number of newly added parameters are trained, enhancing the ability to efficiently fine-tune parameters. LLaMA-7B is used as the pre-trained model, with a weight matrix of... LoRA decomposes it into:

[0051] in, It is a newly added low-rank matrix . It is rank, and the total number of parameters starts from... Reduce to Maintain pre-trained weights No change, only train the two inserted low-rank matrices and Input data After the original weights and adapter The combined effect of these factors yields the output vector. : .

[0052] In this embodiment, the Actor module is responsible for generating the agent's action decisions in the environment. During training, the LoRA parameter is initialized to zero, meaning that in the initial stage, the actor's behavior completely inherits the output of the LLaMA-7B pre-trained model. As training progresses, LoRA gradually adapts to changes in the environment, thereby enhancing the model's task adaptability. To accelerate training convergence, highlighted historical policy data is introduced to ensure that the actor can learn the optimal policy more quickly and improve the stability of policy optimization.

[0053] 3. Training and Reasoning Process Module (1) During training, the agent generates experiential data through interaction with the environment, and this experiential data will be... The data is stored in the playback buffer in the form of [data].

[0054] When the training iterations reach a set threshold, historical highlighted data is sampled from the replay buffer and combined with new data generated by the current strategy for training. Data used each time is discarded from the replay buffer to avoid overfitting and prevent inconsistent data distribution.

[0055] The replay buffer provides a target description corresponding to the highlighted historical strategy as an action cue, a description of the current environmental observation as an observation cue, and the corresponding scalar reward.

[0056] Collect action cues, observation cues, and rewards to provide information to LLaMA-7B, and then use the rewards returned by the environment to fine-tune the LoRA-adapted LLaMA-7B through HiPPO, thereby improving the agent's decision-making ability.

[0057] In this embodiment, to ensure training stability, the Critic and Actor employ different learning rates. The Critic uses a higher learning rate to quickly adapt to new data and provide stable value estimates, while the Actor uses a lower learning rate to ensure smooth policy updates and avoid drastic fluctuations. This ensures both the smoothness and stability of policy updates.

[0058] (2) During the reasoning phase, HiPLM’s design optimizes the efficiency and generalization of the reasoning process. Only the actor module is retained during reasoning, while the commentator module is discarded.

[0059] This is because the commentator is primarily used for value function estimation during the training phase, and is irrelevant to the inference task. This design reduces the computational overhead during inference and improves operational efficiency. The LoRA parameters of the actors fully encode the alignment information between LLMs and the environment during training, and can therefore be directly used during inference. Because HiPPO combines highlighted historical experience data and current data during training, the model in the inference phase has stronger generalization ability and can adapt to different environmental requirements.

[0060] Application Examples 1. Environment Configuration Overcooked Environment: In a 7x7 kitchen environment, the agent's goal is to prepare and deliver tomato salad or tomato-lettuce salad to the starred cells. The challenge lies in the fact that the salad recipes are unknown to the agent. The agent can move within the environment using basic single-step actions such as up, down, left, right, and pause, performing tasks like picking up, placing, cutting, and delivering items by standing next to the corresponding cells. Examples include holding vegetables, placing chopped vegetables on a plate, and delivering items to the target cell. The task provides corresponding ingredients and tools, such as... Figure 3 and Figure 4As shown. In the second environment, an additional distracting ingredient, onion, was added to verify the robustness of HiPLM. The agent needs to explore and learn the correct cooking sequence, using provided verbal actions such as chopping vegetables, getting tomatoes, and going to the cutting board to complete the dish preparation. This environment is partially observable; the agent can only observe objects within a 5×5 grid centered on itself. Correct delivery is rewarded, while incorrect delivery is penalized or wastes time. The reward mechanism is set up so that cutting the correct ingredient earns a reward. The reward is calculated as follows: delivering the correct dish will earn a finishing reward; delivering the wrong item will deduct the reward, indicated as... Rewards will be deducted after each time step, as shown below. .

[0061] VirtualHome Environment: VirtualHome simulates a home environment containing multiple rooms, such as a living room, kitchen, and bedrooms, each with basic items and furniture. The complexity of the environment allows the agent to learn and operate within relatively complex scenarios, placing the agent within this environment. Compared to Overcooked, this environment is more complex and offers a larger action space. The agent interacts with the environment using macro-actions, such as walking to the living room, turning on the TV, and sitting on the sofa. Figure 4 and Figure 5 As shown, in Figure 4 In the first task, Food Preparation, the agent needs to find a cold pancake on a table and heat it using the microwave in the kitchen. Figure 5 In the second task, Entertainment, the agent plans to engage in some entertainment. Therefore, it needs to collect potato chips and milk from the kitchen, bring them to the living room, turn on the TV, sit on the sofa, and enjoy the entertainment. The challenge is that when the agent is holding both the milk and the chips, it lacks the extra hand to operate the TV. Therefore, the agent needs to learn to place at least one item on the nearby coffee table before operating the TV. Both tasks employ a sparse reward setting, where the agent only receives a reward upon completion of the task. Rewards. The environment was also partially observable; the agent could only see items in the current room and not items in other rooms. Observations consisted of a set of Boolean values ​​indicating whether the agent could see relative items, whether those items were close to the agent, and the state of the items, such as whether the TV was on or whether milk was on the coffee table. All experiments were conducted on a server equipped with an NVIDIA A40 40GB GPU.

[0062] 2. Model parameter settings

[0063] 3. Implementation steps of the dynamic normalization module (1) Perform text preprocessing on task objectives (such as “making tomato salad”) and candidate action prompts (such as “picking up tomatoes” and “cutting lettuce”) to remove redundant symbols; (2) Use a pre-trained BERT model to generate embedding vectors of the same dimension for the target and the action; (3) Calculate the vector cosine similarity to obtain the relevance score between the action and the task, which is used as the dynamic weight; (4) Based on the Token-level log probability output by the large language model, the probability distribution is adjusted by dynamic weighting to obtain the normalized action probability.

[0064] 4. Training Process Implementation Steps (1) Initialization: Load the LLaMA-7B model and freeze the weights, initialize the LoRA low-rank matrix (parameters are zero), and initialize the Critic MLP module (random initialization). (2) Experience collection: The agent performs actions in the environment and records experience data (s t , a t , r t , s t+1 ), stored in the playback buffer; (3) Highlighted sample sampling: When the training rounds reach the threshold, samples that meet the following conditions are selected from the buffer: reward r t >0.5, similarity to the current state >0.7 (based on the cosine distance of the state feature vector), and dominance estimate >0 (Critic evaluation value); (4) Parameter update: Critic Update: Calculate the value loss (mean squared error) using the highlighted sample and the current sample, and optimize the MLP parameters through gradient descent; Actor Update: Based on the advantage value output by the Critic, the policy loss is calculated by combining the TV distance penalty, and only the low-rank matrix parameters of LoRA are updated; Learning rate adjustment: When the TV distance between policies is greater than ξ / 2, the Actor's learning rate is halved to ensure update stability; Iterative training: Repeat steps 2-4 until the agent achieves a task completion rate of >90% in 100 consecutive rounds.

[0065] 5. Implementation steps of the reasoning process (1) Freeze the LLaMA-7B weights and Critic module, and keep only the LoRA parameters of the Actor module; (2) Input environmental observations (such as "there are tomatoes and plates at the current location"), and generate action probabilities through the dynamic normalization module; (3) Select the action with the highest probability to perform and complete the interaction with the environment.

[0066] To investigate the impact of online fine-tuning with HiPPO on the capabilities of large language models, a HiLPM model trained using dynamic normalization techniques in a Virtualhome environment was evaluated based on widely used NLP benchmark sets.

[0067] The models trained for food preparation and entertainment scenarios are named HiLPM-FP and HiLPM-E, respectively. This environment aims to simulate everyday human activities and situations, realistically reflecting the language model's ability to handle complex and dynamic environments. To further validate their performance on different tasks, four challenging multi-dimensional NLP benchmark sets were selected: ARC_C, HellaSwag, PIQA, and MMLU.

[0068] After conducting these benchmark tests, the zero-shot performance results shown in Table 1 demonstrate that the HiLPM model, fine-tuned by HiPPO, did not show significant degradation in zero-shot performance after environment alignment, and even exhibited slight improvements in some tasks. The HiLPM-FP model showed considerable progress on tasks such as ARC_C, HellaSwag, PIQA, and MMLU. Specifically, HiLPM-FP improved performance in ARC_C and MMLU, reaching 0.45 and 0.38 respectively, representing improvements of 7.1% and 15.2% compared to the original LLaMA-7B baseline. HiLPM-E, on the other hand, performed stably on multiple tasks, showing slightly higher accuracy (0.59) on HellaSwag, which requires complex physics reasoning, representing a 3.5% improvement over the original LLaMA-7B baseline.

[0069] Table 1. Zero-shot performance on the language model evaluation platform.

[0070] These results demonstrate that the HiPPO online fine-tuning method successfully improves the model's language understanding and reasoning capabilities across multiple tasks by fine-tuning the model in a dynamic virtual environment. While this fine-tuning method did not significantly improve performance across all tasks, it showed a clear advantage in certain specific tasks, particularly those requiring common-sense and physical reasoning. This indicates that, through appropriate environment alignment and fine-tuning, language models can better adapt to complex and diverse real-world tasks, thereby enhancing their practical application capabilities across various tasks.

[0071] Through the above implementation scheme, the highlight playback strategy optimization framework constructed in this disclosure achieves efficient optimization of large language models in complex decision-making tasks. Its significantly improved action effectiveness and sample utilization rate have multi-dimensional and profound value in real-world scenarios.

[0072] From the perspective of the universality of technological applications, this framework breaks through the bottleneck of "theoretical suggestions being disconnected from actual implementation" in decision-making tasks for large language models. The improved effectiveness of actions means that the model can more accurately adapt to the constraints of real-world environments, reduce resource waste caused by ineffective outputs, and transform large language models from "text interaction tools" to "decision-making entities with practical operational capabilities," thus clearing key obstacles for their implementation in various scenarios requiring dynamic responses.

[0073] Optimizing sample utilization is directly related to the economic viability and feasibility of technology implementation. In practical applications, whether in virtual or physical environments, acquiring interactive data often requires significant time, computing power, and material costs. This disclosure reduces reliance on real-time data by efficiently reusing historical experience, shortening the model adaptation cycle to new scenarios and reducing unnecessary environmental interactions. This makes it possible to apply large language models in resource-constrained scenarios, further lowering the barrier to technology adoption.

[0074] From an industry development perspective, this framework retains the original capabilities of large language models, enabling them to enhance decision-making and execution capabilities while maintaining their general knowledge and reasoning advantages. This provides core support for the construction of cross-domain intelligent systems. It drives the leap of artificial intelligence technology from single language processing to a full-chain capability encompassing "understanding-decision-execution," accelerating the integration of intelligent technology with the real economy, injecting new momentum into the intelligent upgrading of various industries, and helping to improve overall social productivity and service quality.

[0075] Furthermore, the stable and efficient strategy optimization achieved by this framework lays the foundation for the application of large language models in more complex scenarios that are closer to human life, and promotes the transition of artificial intelligence technology from the laboratory to actual production and life. Its value lies not only in the technological breakthroughs, but also in promoting the true integration of intelligent systems into the operation of society, bringing tangible convenience and innovation to human life.

[0076] The above technical solutions are merely exemplary embodiments of the present invention. For those skilled in the art, based on the application methods and principles disclosed in the present invention, it is easy to make various types of improvements or modifications, and not limited to the methods described in the specific embodiments of the present invention. Therefore, the methods described above are merely preferred and not restrictive.

Claims

1. A large language model framework for optimizing highlighting and playback strategies in AI econometrics scenarios, characterized by: During the training phase: the Actor-Critic framework is adopted, sharing the frozen large language model, and integrating the low-rank adapter LoRA to achieve efficient parameter fine-tuning; specifically including: The large language model serves as the foundation of the framework; The Actor module is used to generate action decisions for agents in the environment, and consists of a frozen large language model plus LoRA. The Critic module consists of an additional multilayer perceptron added after the last Transformer block of the large language model; the Critic module is jointly trained with the highlighting history strategy data, and the TV distance is used as the reward constraint. A replay buffer is used to store experiential data generated by the agent through its interaction with the environment; The replay buffer provides highlighted historical policy data, including action cues and observation cues, as well as corresponding scalar rewards, which are then fed into the large language model. LoRA fine-tunes the adapted large language model to improve the agent's decision-making ability. During the inference phase, only the Actor module is retained, while the Critic module is discarded.

2. The framework according to claim 1, characterized in that, The input to the large language model includes action cues and observation cues, wherein each action cue... A sequence consisting of multiple tokens Composition, in which This indicates the length of the action cue; the length of the action cue can vary; in each step, the observed cue... Possibly related to the currently selected action Related, through effective actions Concatenation is used as input to a large language model; The large language model generates token-level probabilities, which are used to calculate the probability of action cues. The token-level probabilities are as follows: Typically, large language models provide scores based on the log-likelihood of each token, i.e., the log probability. ; The action is normalized using the Softmax function to obtain the policy. : 。 3. The framework according to claim 2, characterized in that, Also includes: The dynamic normalization module is used to dynamically adjust the normalization weights to adjust the generation probability of actions, thereby increasing the probability of action prompts that are relevant to the task and reducing interference from actions that are irrelevant to the task.

4. The frame according to claim 3, characterized in that, In the dynamic normalization module, the joint probability is calculated using the following formula: in, Action prompts The number of words in Action prompts Dynamic weights, , The correlation between actions and the current task objective: In the formula, Representative actions With observation tips The similarity between them; The similarity calculation uses the BERT model, which transforms each action cue and task objective into a high-dimensional vector representation. Then, the cosine similarity between these vectors is calculated to quantify their relevance. The formula for calculating cosine similarity is: in, and These represent the embedding vectors for action cues and task objectives, respectively. This represents the dot product operation. The norm of a vector.

5. The frame according to any one of claims 1-4, characterized in that, The LLaMA-7B is used as the basic model for the framework.

6. The frame according to claim 5, characterized in that, In the Actor module, the function of LoRA is represented as follows: Let LLaMA-7B be the pre-trained model, and the weight matrix be... ; LoRA breaks it down into: in, For the newly added low-rank matrix, ; For rank, the total number of parameters starts from Reduce to ; Maintain pre-trained weights No change, only train the two inserted low-rank matrices and ; Input data After the original weights and adapter The combined effect of these factors yields the output vector. : 。 7. The framework according to claim 1, characterized in that, The training process of the framework includes the following steps: The agent generates experiential data through interaction with the environment, and this experiential data is stored in a replay buffer; When the training rounds reach the set threshold, the highlighted historical experience is sampled from the replay buffer and combined with the new data generated by the current strategy for training. In this step, the data used each time is discarded from the replay buffer. The replay buffer provides the target description corresponding to the highlighted historical strategy as an action cue, the description of the current environment observation as an observation cue, and the corresponding scalar reward. Collect action cues, observation cues, and rewards to provide information to the large language model, and then use the rewards returned by the environment to fine-tune the adapted large language model through LoRA, thereby improving the agent's decision-making ability; To ensure training stability, Critic and Actor employ different learning rates: Critic uses a higher learning rate to quickly adapt to new data and provide stable value estimates, while Actor uses a lower learning rate to ensure smooth policy updates.

Images