Large language model reasoning method based on time difference learning and rule enhancement

Abstract

The application relates to the field of natural language processing and decision intelligence, and particularly relates to a large language model reasoning method based on time series difference learning and rule enhancement, which is widely applied to automatic planning, intelligent question answering, embodied intelligence and the like. The method comprises task trajectory sampling, domain knowledge induction, domain rule extraction and large language model reasoning based on rule enhancement. For test task data, the most relevant historical task is matched based on vector retrieval, and a domain rule set corresponding to the task is obtained. The rule is rewritten in natural language by the large language model itself, so that the rule is more interpretable and adaptable. Finally, the optimized rule set is integrated into the large language model reasoning prompt text, so as to optimize the reasoning quality and stability.

Description

Technical Field

[0001] This application relates to fields such as natural language processing and decision intelligence, and in particular to a large language model reasoning method based on temporal difference learning (TD learning) and rule augmentation. This method can be widely applied to scenarios such as automatic planning, intelligent question answering, and embodied intelligence. Background Technology

[0002] In the field of agent reasoning and decision-making, multi-step reasoning ability is crucial for improving task processing efficiency and accuracy. Existing agent decision-making methods typically employ reinforcement learning, optimizing decision strategies through interactive trial and error. However, reinforcement learning methods heavily rely on large amounts of environmental interaction data, resulting in high training costs and limited generalization ability when dealing with new tasks. In recent years, the research direction of applying Large Language Models (LLMs) as agents to multi-step reasoning and decision-making tasks has attracted widespread attention. These models are pre-trained on large-scale text data, possessing rich world knowledge and reasoning capabilities, enabling them to perform complex multi-step decision-making tasks in real-world environments. Their decision-making paradigm can be summarized as the language interaction between the agent and the environment: the LLM, as the agent, reasons appropriate actions based on the text descriptions provided by the environment, combined with existing knowledge, and adjusts its strategy based on environmental feedback. In this process, LLM relies on internal knowledge and memory mechanisms for long-term reasoning and complex planning, and can combine methods such as in-context learning (ICL) and retrieval-augmented generation (RAG) to introduce external knowledge and optimize reasoning through knowledge enhancement.

[0003] However, existing knowledge augmentation methods still have limitations in terms of knowledge utilization efficiency, generalization ability, and inference optimization. For example, context-based learning methods guide large language models to reason by injecting examples into prompts; however, this method is limited by the size of the context window, making it difficult to store and reuse knowledge long-term. Furthermore, some studies inject domain knowledge into large language models through model fine-tuning, such as using procedural reward models to optimize inference strategies. However, these methods typically require large amounts of high-quality labeled data, resulting in high training costs and difficulty in adapting to closed-source large language models, thus limiting their practical applications.

[0004] Based on this, this invention applies a knowledge augmentation method based on temporal difference learning and rule induction to optimize multi-step reasoning in large language models, thereby improving the reasoning decision-making level of LLM. This invention effectively solves the problems of existing knowledge augmentation methods, such as reliance on high-quality positive sample data and the difficulty in balancing external knowledge injection density and computational cost. It can be widely applied to scenarios such as automated planning, intelligent question answering, and embodied intelligence. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention aims to optimize multi-step reasoning in large language models based on a knowledge enhancement method using temporal difference learning and rule induction. This method is applicable to scenarios where tasks can be decomposed into several sub-goals. In such tasks, the agent can measure task progress based on the achievement of decision-making sub-goals and optimize the reasoning process by combining rule enhancement.

[0006] The specific technical solution is as follows: A large language model reasoning method based on Temporal Difference Learning (TDLearning) and Rule Augmentation, including the following steps;

[0007] (1) Task trajectory sampling: Based on the base large language model, multiple rounds of random sampling are performed on the training task dataset to generate diverse task inference trajectories; further, the sampled inference trajectories are filtered to retain only the valid trajectories.

[0008] (2) Domain knowledge induction: Extract action semantics from the action steps of all valid reasoning trajectories and map them to a unified action space; classify the reasoning trajectories according to the domain to which the task belongs, aggregate all trajectories in the same task domain, and extract the general execution pattern in the domain; construct a domain knowledge graph based on the aggregated trajectory set, with deparameterized action semantics as nodes and action semantic dependencies in the trajectory as directed edges, to form a knowledge structure in the task domain.

[0009] (3) Domain rule extraction: The temporal difference method is used to learn the contribution of different actions to task progress until it converges, and PageRank is combined to evaluate the global influence of actions in the domain knowledge graph. By integrating local contribution and global importance, key task execution rules are extracted and action dependencies are formed, thereby establishing a domain rule set that can be used to optimize LLM inference.

[0010] (4) Rule-enhanced large language model reasoning: For test task data, the most relevant historical tasks are matched based on vector retrieval, and the domain rule set of the corresponding task is obtained; the rules are rewritten in natural language by the large language model itself to make them more interpretable and adaptable; finally, the optimized rules are integrated into the large language model reasoning prompt text to optimize reasoning quality and stability.

[0011] Step (1) Specifically, the training task dataset Each task instance (m, o) consists of a task domain m and task-specific information o. The task domain m represents the category to which the task belongs; the task-specific information o contains the input information for the task. The LLM needs to reason about these task instances, generating possible task reasoning trajectories. These trajectories contain the states at multiple time steps, the actions performed, and the corresponding task progress, describing how the large language model advances the task progress during multi-step reasoning. The set of sampled trajectories can be formally represented as:

[0012]

[0013] τ={(s1,a1,r1),(s2,a2,r2),…,(s T ,a T ,r T )},

[0014] Here, τ represents the complete execution trajectory inferred by the large language model on a specific task instance (m,o), which includes all decision steps from the initial state to the final result. To ensure the diversity and rationality of the sampled trajectories, temperature scaling and Nucleus sampling (Top-p sampling) are used for trajectory sampling.

[0015] Furthermore, in the multi-step decision-making process, for each decision step, the agent outputs the current action based on the current environment information. Because the actions output based on the large language model have uncertainty, the quality of the sampled inference trajectory is improved through data filtering. The filtering process includes the following key steps: eliminating illegal actions: ensuring that each action in the sampled trajectory conforms to the basic logical rules of the task environment; removing invalid actions: avoiding the large language model from generating redundant inference paths. Step (2) Specifically, since the large language model may express the same inference steps in different ways, it is necessary to map the actions in the sampled trajectory to a unified semantic space. Each decision action a t From action semantics and motion parameters θ t Composition. Among them, action semantics. This represents a general operation that spans different task scenarios, while action parameters refer to specific task instances. To aggregate task trajectories within the same domain, action parameters are removed, mapping all actions of the same category to a unified action space, retaining only the action semantics. Then, based on the task domain m, all trajectories within the current task domain are aggregated to obtain T. m Then, a directed acyclic graph G is constructed for the task domain m. m = (V, E). Here, the node set represents the deparameterized action semantics in the task. The edge set represents the action dependencies in the trajectory. If an action... Task execution usually precedes another action. If this happens, then there will be a path in the graph from... arrive directed edges It reflects the execution logic of actions during the reasoning process.

[0016] Step (3) specifically involves using temporal difference learning to calculate the local contribution of each action and combining it with a link analysis-based calculation method to evaluate the global influence of the action on the task graph. Through this process, task execution rules are constructed, and a structured set of domain knowledge is formed. During task execution, different actions contribute unevenly to the final goal. Some key actions may play a decisive role in task progress, while the influence of other actions may be smaller. Therefore, it is necessary to calculate the action TD contribution Q, that is, the degree of influence of a certain action on the future task progress in a specific state. To this end, the temporal difference error method is used to update the TD contribution Q, thereby ensuring that the Q value can correctly reflect the contribution of the action. Furthermore, the eligibility trace is used to improve the stability of learning. Its iterative update process is as follows:

[0017]

[0018] Where, δ t Represents the timing difference error (TD Error), used to measure the current action. Whether the contribution is as expected; p t γ is the immediate contribution, which is the incremental progress of the task directly brought about by the action; γ is the discount factor, used to control the weight of the influence of future states on the current decision. The value of γ is between [0,1]. A larger γ makes the model pay more attention to long-term benefits, while a smaller γ makes the model more inclined to short-term benefits; It represents the TD contribution level in the next stage, indicating the potential future benefits; It represents the TD contribution of the current action, which is continuously updated during training; α is the Eligibility Trace Value, used to record the degree to which an action contributes to the progress of the task; α is the learning rate, which controls the step size of Q-value updates; and λ is the Trace Decay Factor.

[0019] In addition to calculating local contributions, it is necessary to evaluate the global influence of each action in the task graph. During task execution, some actions may be core steps on the critical path, while others may only be auxiliary operations. Therefore, it is necessary to measure whether an action plays a critical role in the entire task process. To this end, the PageRank calculation method is introduced to evaluate the global influence of actions in the domain knowledge graph. The TD contribution and PageRank influence are combined to calculate the comprehensive action contribution, form the task execution rules, and store them in the domain knowledge set.

[0020]

[0021] in, The final contribution of the representative action, taking into account its local benefits. and global influence β is a weighting parameter used to adjust the relative weights of TD contribution and PageRank influence.

[0022] Step (4) specifically involves retrieving the most relevant training task when a test task arrives, in order to obtain rule knowledge applicable to the current task. To this end, a vector retrieval method is used, which calculates the current task description h... i With training task description h j The cosine similarity between the training tasks is used to find the most similar training tasks and obtain their rule set.

[0023]

[0024] Here, Embed(·) represents the text embedding model, used to convert text task descriptions into high-dimensional vector representations, allowing the similarity between different tasks to be measured mathematically; cos(·,·) is the cosine similarity calculation function, used to measure the similarity between two task description vectors. Then, the K rules with the highest contribution are selected from the retrieved task rule set and used to optimize the reasoning ability of the large language model. The key to this process is ensuring that the large language model only refers to the optimal rules for reasoning optimization, avoiding the introduction of irrelevant or inefficient rules that interfere with the reasoning results. The calculation method for selecting rules is as follows:

[0025]

[0026] Among them, Π mIt is the complete set of rules corresponding to task domain m; The contribution of an action represents its impact on task progress; the contribution calculation combines temporal difference learning and PageRank influence; TopK(·) represents the number of actions ranked by contribution. Sort the rules and select the top K rules that are most helpful for task completion. Recall the most relevant set of rules. These represent the rules most suitable for the current testing task. Since these rules are derived from the execution data of the training task, they not only have high reliability but are also directly applicable to the testing task, improving the reasoning ability of the large language model. Few-Shot Learning is employed to optimize the selected rules using natural language based on the internal knowledge of the LLM, making them more interpretable and applicable to different task scenarios.

[0027] Finally, the optimized rules are integrated into the LLM's inference hints, enabling it to make more accurate decisions in test tasks.

[0028] The features and beneficial effects of this invention are:

[0029] (1) This invention utilizes temporal difference learning and rule induction to perform LLM knowledge enhancement reasoning, making full use of the large amount of non-positive sample information generated by the training task. By sampling the training task multiple times and using temporal difference learning, the value contribution of state-action to achieving the decision goal is calculated from the sampled samples, and a generalizable external knowledge base is constructed.

[0030] (2) This invention is based on LLM self-boosting rule-enhanced context learning, which uses LLM itself to extract and generalize rules based on state-action contributions. This rule-enhanced context learning effectively improves the information entropy of context learning, achieving a balance between instance-based prompting learning and model fine-tuning, and balancing the density of external knowledge injection and computational cost.

[0031] (3) This invention can adapt to various types of task scenarios and support the automatic summarization and optimization of knowledge in different task domains, thereby enhancing the generalization ability and adaptability of the reasoning process, enabling it to be widely applied in multiple application scenarios such as complex task planning, intelligent reasoning and automatic decision-making. Attached Figure Description

[0032] Figure 1 This is a framework diagram of a large language model inference method based on temporal difference learning and rule reinforcement.

[0033] Figure 2 This is a flowchart illustrating a large language model inference method based on temporal difference learning and rule reinforcement.

[0034] Figure 3This is a comparison chart of model inference performance. Detailed Implementation

[0035] This invention proposes a multi-step decision optimization method for large language models based on rule enhancement. First, effective inference trajectories are sampled and filtered, action semantics are summarized, and a task-domain knowledge graph is constructed. Then, action contributions are evaluated based on temporal difference learning, and key task rules are extracted. Finally, relevant tasks are retrieved during inference, and LLM self-optimizing rules are incorporated with textual prompts to improve inference quality and stability.

[0036] The specific implementation steps of the present invention are as follows: Figure 1 As shown, it mainly includes the following four steps:

[0037] (1) Task trajectory sampling: Based on the base large language model, multiple rounds of random sampling are performed on the training task dataset to generate diverse task inference trajectories; further, the sampled inference trajectories are filtered to retain only the valid trajectories.

[0038] Specifically, the training task dataset Each task instance (m, o) consists of a task domain m and task-specific information o. The task domain m represents the category to which the task belongs; the task-specific information o contains the input information of the task. For each training data sample, N sampling steps at time T are performed using LLM. The inference trajectory structure obtained from each sampling includes the state, executed actions, and corresponding task progress at multiple time steps:

[0039] τ={(s1,a1,r1),(s2,a2,r2),…,(s T ,a T ,r T )},

[0040] The set of sampling trajectories for all training data can be formally represented as:

[0041]

[0042] To enhance sampling diversity and enable LLM to explore various task execution methods, this method employs a stochastic decoding strategy to avoid generating single-pattern trajectories. Specifically, it combines temperature scaling and nucleus sampling (top-p sampling) to control the exploration range during inference, thereby generating sufficiently diverse trajectories while avoiding low-probability and unreasonable inference paths.

[0043] Temperature regulation adjusts the steepness of the model's output action probability distribution, enabling the sampling process to explore new paths while maintaining reasonable decision stability. Its generation process is as follows:

[0044]

[0045] Among them, z a The score of action 'a' in the model output is represented by δ, which is a temperature parameter that determines the randomness of sampling. Intuitively, the temperature parameter δ controls the model's caution when choosing the next action: when δ is large, the probabilities of all possible actions become closer, and the model has a strong exploratory ability, allowing it to try different inference paths; when δ is small, large language models tend to choose high-probability actions, thereby reducing the uncertainty of inference and improving the stability of the inference results.

[0046] To further optimize the decoding strategy, Nucleus sampling (Top-p sampling) is used to ensure that the large language model selects only from the most representative set of candidate actions, avoiding sampling of extremely unreasonable actions. Its calculation method is as follows:

[0047]

[0048] in, The set of candidate actions representing the top p probability quality of the large language model ensures that the large language model only selects from the most probable actions and does not randomly select options with extremely low probability. It is an indicator function, indicating that it only applies when... Only actions within this range will be sampled. This strategy ensures that the actions sampled by the large language model only include the most representative candidate set, while still retaining a certain degree of exploratory capability.

[0049] Furthermore, since the trajectory sampled by LLM may contain illegal or invalid actions, the data needs to be filtered after sampling to retain only valid paths. All actions can be divided into three categories: (1) Illegal actions: Actions that do not belong to the set of actions allowed by the environment. (2) Legally invalid actions: Actions that can be executed but do not contribute to the progress of the task, i.e., r t =r t-1 (3) Legal and valid actions: actions that are executable and contribute to the completion of the task, i.e., r t >r t-1 Because illegal actions themselves do not affect the state transition of the current task, they can be removed. Specifically, for each final progress r... T For trajectory samples τ > 0, all illegal actions are removed, and only the trajectories formed by legal actions are retained:

[0050] τ′={(s′1,a′1,r′1),(s′2,a′2,r′2),…,(s′ T′ ,a′ T′ ,r′ T′ )},where r′ T′ >0,

[0051] Where T′≤T, and T is the number of time steps of the original sampling trajectory τ.

[0052] (2) Domain knowledge induction: Extract action semantics from the action steps of all valid reasoning trajectories and map them to a unified action space; classify the reasoning trajectories according to the domain to which the task belongs, aggregate all trajectories in the same task domain, and extract the general execution pattern in the domain; construct a domain knowledge graph based on the aggregated trajectory set, with deparameterized action semantics as nodes and action semantic dependencies in the trajectory as directed edges, to form a knowledge structure in the task domain.

[0053] Specifically, in a multi-step decision-making task, each decision action a t From action semantics and motion parameters θ t Composition. Among them, action semantics. This represents a general operation across different task scenarios, while action parameters refer to specific task instances. To aggregate trajectories under the same task category, action parameters are removed, mapping all actions of the same category to a unified action space, retaining only the action semantics. Further, action a is defined. t Progress increment p in trajectory τ t =r t -r t-1 r t r represents the progress measure at the current time t. t-1 This is the progress measure for the previous moment. As a component of the trajectory, the filtered set of sampled trajectories is obtained. Each trajectory τ filtered for:

[0054]

[0055] Because action semantics share logical commonalities within the same domain, this representation method can effectively aggregate decision trajectories from different task instances and be used for knowledge induction or strategy optimization. Based on the task domain m, T is obtained by aggregating all trajectories within the current task domain. m Then, a directed acyclic graph G = (V, E) is constructed for the task domain m, where the node set and edge set are defined as follows:

[0056]

[0057] in, Action semantics Average progress contribution across different trajectories:

[0058]

[0059] in, This indicates that it contains action semantics. The set of trajectories, i.e. Domain Knowledge Graph G m It can accurately characterize the execution process of the task domain m and provide domain knowledge support for subsequent action contribution calculations and rule extraction.

[0060] (3) Domain rule extraction: The temporal difference method is used to learn the contribution of different actions to task progress until it converges, and PageRank is combined to evaluate the global influence of actions in the domain knowledge graph. By integrating local contribution and global importance, key task execution rules are extracted and action dependencies are formed, thereby establishing a domain rule set that can be used to optimize LLM inference.

[0061] Specifically, after constructing the domain knowledge graph G corresponding to the task domain m... m Next, a domain rule set Φ is derived from this, aiming to capture the rule structure across specific tasks within the task domain m:

[0062] Φ={(φ i ,ψ i ,θ)∣i=1,…,|Φ|},

[0063] Where, φ i Antecedent; ψ i A causal action (Consequence) indicates that, in order to satisfy φ i The preferred actions to take under given conditions; θ represents the contribution of causal consequences to task completion. Specifically, it is necessary to calculate the semantics of each action. Contribution to task progress. A temporal difference (TD)-based method is used to estimate the local contribution of different actions to task progress. Simultaneously, the PageRank method is introduced to calculate the global influence of actions within the domain knowledge graph, ultimately obtaining a comprehensive estimate of their contribution to task progress.

[0064] In the mission trajectory, Representing action semantics The contribution to mission progress, i.e., the TD contribution. Estimation is performed using temporal difference, where the state transitions in the trajectory are related to the immediate reward Δp. t for:

[0065]

[0066] Δp t =p t+1 -p t ,

[0067] Based on this, the TD error is calculated as follows:

[0068]

[0069] Where γ∈[0,1] is the discount factor, used to balance the influence of current rewards and future rewards. To improve the stability and convergence speed of learning, an eligibility trace is introduced, and TD(λ) is used for Q-value updates. The update rules are as follows:

[0070]

[0071] Where λ∈[0,1] is the eligibility trace decay factor, used to control the influence of historical information; α is the learning rate, determining the step size for Q-value updates. By accumulating past TD errors, this method reduces estimation variance and improves adaptability to long-sequence tasks while balancing short-term rewards and long-term contributions. To effectively control computational complexity, a random trajectory sampling strategy is adopted, based on the N-axis of the entire graph. total A complete path, in each round only N samples TD(λ) calculation is performed on each sampling path, and multiple iterations are conducted until convergence. Specifically, let the Q-value of each action semantic be after the k-th iteration. If, in several consecutive iterations, the magnitude of the Q-value change for all actions is below a certain threshold ∈, then the Q-value is considered to have converged.

[0072]

[0073] Here, ∈ takes values ​​ranging from 10e2 to 10e3 to ensure the stability of the estimated value. To prevent local oscillations from being misjudged as convergence, this condition must be met in multiple consecutive iterations.

[0074] In addition to assessing the direct contribution of TD computation actions to task progress, it is also necessary to evaluate the global influence of these actions within the entire domain knowledge graph. Domain Knowledge Graph G m Some actions in a domain knowledge graph may play a crucial role in task completion due to their high frequency of occurrence or connection to multiple critical paths. Therefore, the PageRank calculation method is introduced to evaluate the importance of actions in the domain knowledge graph.

[0075]

[0076] in, It is an action The PageRank value represents the importance of an action in the domain knowledge graph; d is the damping factor, which controls the degree of dependence on other actions during a random walk. The core idea of ​​the PageRank calculation method is: if an action... If an action is depended upon by several important preceding actions, it is itself important. Therefore, PageRank assigns higher scores to actions that act as bridges in the task execution process.

[0077] By combining TD contribution and PageRank influence, a comprehensive action contribution is calculated to form task execution rules, which are then stored in the domain knowledge base.

[0078]

[0079] in, The final contribution of the representative action, taking into account its local benefits (TD contribution). and global influence β is a weighting parameter used to adjust the relative weights of TD contribution and PageRank influence. Scores are based on contribution. Sort all actions to obtain the domain rule set:

[0080]

[0081] in, Indicates semantic action The set of parent action nodes in the domain knowledge graph, namely:

[0082]

[0083] Where E m Let G = (V, E) be the set of directed edges representing dependencies in the task flow. Specifically, let Π be the set of domain rules. m It is a set of actions arranged in descending order of contribution in the task domain m. Each rule contains a semantic action, a contribution estimate, and a prerequisite dependency action.

[0084] (4) Rule-enhanced large language model reasoning: For test task data, the most relevant historical tasks are matched based on vector retrieval, and the domain rule set of the corresponding task is obtained; the rules are rewritten in natural language by the large language model itself to make them more interpretable and adaptable; finally, the optimized rules are integrated into the large language model reasoning prompt text to optimize reasoning quality and stability.

[0085] Specifically, rule sets based on specific task domains Π mFurthermore, a rule-reinforcement-based context learning mechanism is introduced, which uses rule retrieval enhancement in the reasoning stage to provide LLM with referable rule knowledge, thereby optimizing its reasoning process and improving interpretability and generalization ability.

[0086] To construct rule knowledge that can be used for testing tasks, we first utilize the VectorRetrieval mechanism to retrieve rules from the training task description set. Retrieve the task most relevant to the current test task and determine its corresponding task domain m. Calculate the initial description h of the current task. i With training task description Embedding similarity between (Cosine)

[0087] (Similarity) Find the most similar training task and its corresponding task domain m:

[0088]

[0089] After retrieving similar training task descriptions h m Then, its corresponding task domain m is obtained from the rule set Π. m Select the K rules with the highest contribution:

[0090]

[0091] Where TopK(·) represents the order of contribution. Sort the rules and select the top K. Further, summarize and optimize the selected rules. Each rule... Formalized as an Antitecedent-Consequence rule:

[0092] Antecedent[Conditions]→Consequence[Actions],

[0093] Among them, the prerequisites As part of Antecedent; As the Consequence part, to enhance the interpretability of the rules, we base them on few-shot learning and utilize LLM to perform Domain Rule Self-Refinement on the Antecedent-Consequence rules, making them more consistent with natural language expressions. Let the initial rule be:

[0094] R=(Antecedent A, Consequence C).

[0095] Define LLM decoding as Optimization under the few-shot learning paradigm:

[0096]

[0097] in, It is a few-shot learning example set, including high-quality rule rewrite samples, which makes the optimized rule R′ more interpretable and can be integrated into the inference hint structure of LLM as interpretable task experience to guide the inference process:

[0098]

[0099] Among them, s t This represents the state information of the agent input to the current test task, a. t This indicates the next action that the agent will output in the domain rule set R′.

[0100] Through the above technical solutions, this invention significantly improves the adaptability and stability of large language models in multi-step reasoning and decision-making tasks by introducing a rule-based learning-based knowledge enhancement mechanism. By sampling and filtering task trajectories, this method can generate diverse task reasoning paths and construct a structured domain knowledge graph using trajectory aggregation and knowledge induction methods, thereby extracting general task knowledge. Combining temporal difference learning and link analysis computation, this method can accurately evaluate the local contribution and global influence of each action and automatically summarize the rule knowledge that optimizes the reasoning process of the large language model. Compared with traditional direct reasoning methods for large language models, this application can not only use vector retrieval to match the most relevant rules in test tasks, but also perform rule induction through few-shot learning, making the reasoning results more interpretable and generalizable. Furthermore, by integrating the optimized rules into the reasoning prompts of the large language model, this method effectively reduces the incidence of reasoning errors and illusions, improving the credibility of task decisions. Overall, this invention introduces a new technical method into multi-step reasoning tasks of large language models, promoting the development of knowledge-enhanced intelligent decision optimization schemes, and can be widely applied to task scenarios requiring complex reasoning capabilities, such as automatic planning, intelligent question answering, and embodied intelligence.

[0101] Example 1: Rule-enhanced Family Task Reasoning Execution

[0102] In this embodiment, the reasoning method is applied to a family task environment with an embodied agent, where the agent needs to complete complex multi-step tasks through text interaction. Taking "putting two soap packets into the trash can" as a representative task, the specific application flow of the method is demonstrated:

[0103] 1. Task trajectory sampling phase:

[0104] (1) During the training phase, the base big language model (LLM) samples task trajectories related to “item sorting and processing” in home scenarios such as kitchen and bathroom, such as “putting dirty plates into the sink” and “throwing food into the trash can”.

[0105] (2) Each trajectory contains state-action pairs at multiple time steps, including textual state descriptions (e.g., “You are standing at the bathroom door”) and model-generated actions (e.g., “Open the door”, “Enter the room”, “Pick up the soap”).

[0106] (3) Temperature control and Top-p sampling mechanism are used to ensure trajectory diversity, and illegal and redundant actions are eliminated by trajectory filtering algorithm to retain valid trajectories.

[0107] 2. Domain knowledge summarization and rule extraction:

[0108] (1) All trajectory actions are mapped to a unified semantic space. For example, the actions “open the refrigerator” and “open the trash can” are all unified as “OPEN_CONTAINER” semantics.

[0109] (2) Construct a domain knowledge graph and extract action dependencies, such as “OPEN_CONTAINER” → “PICK_OBJECT”.

[0110] → "PUT_OBJECT".

[0111] (3) Calculate the TD contribution of each action based on temporal difference learning, and combine it with PageRank to calculate the global influence, forming rules:

[0112] Rule 1: If the target container is closed, then execute OPEN_CONTAINER (TD contribution = 0.35, PageRank = 0.42). Rule 2: If the item is reachable, then execute PICK_OBJECT (TD contribution = 0.31, PageRank = 0.30).

[0113] 3. Rule-enhanced reasoning stage:

[0114] (1) In the test task “Put two soap packages into the trash can”, the system uses vector retrieval to match the “Household Scene - Garbage Disposal” task domain and recalls the high contribution rule set.

[0115] (2) Prompts are given in natural language form through LLM rewriting rules:

[0116] "Please check if the trash can is open first. If necessary, you should open it first."

[0117] "Please get close to the target location before picking up the target item."

[0118] (3) The optimized rule hints are embedded into the reasoning context of the model, which significantly improves the rationality of its actions and the efficiency of its completion.

[0119] Example 2: Optimization of Scientific Experiment Process Reasoning Based on Rule Enhancement

[0120] In this embodiment, the reasoning method is applied to task reasoning and operation planning in a scientific experiment scenario. The agent needs to complete a causal reasoning task based on observation results in an environment equipped with various experimental devices. Taking "determining whether an iron block has thermal expansion properties" as a representative task, the specific application process of the method is demonstrated:

[0121] 1. Task trajectory sampling phase:

[0122] (1) During the training phase, the base large language model (LLM) samples trajectories on related tasks such as “measuring the physical properties of metals”, such as “measuring the length of an aluminum rod before and after heating”, “monitoring the heating process of copper metal with a thermometer”, and “recording the trend of copper block size change with temperature”.

[0123] (2) Each trajectory contains multiple state-action pairs at multiple time steps, including textual state descriptions (such as "You are standing in front of the lab bench", "The thermometer shows 30 degrees Celsius") and model-generated actions (such as "Measure length").

[0124] "Heating metal" and "recording data").

[0125] (3) Diverse experimental strategy paths are generated through temperature control and Top-p sampling, and the trajectories are screened to eliminate redundant or logically erroneous operations, retaining only the valid reasoning paths.

[0126] 2. Domain knowledge summarization and rule extraction:

[0127] (1) Map all trajectory actions to a unified semantic space, such as unifying actions like “measuring metal dimensions” and “reading temperature” into semantic actions like “MEASURE_OBJECT” and “READ_TEMPERATURE”.

[0128] (2) Construct a knowledge graph in the field of “experimental study of thermal properties of metals” and explore the dependency order between typical experimental steps, such as “MEASURE_OBJECT” → “HEAT_OBJECT” → “MEASURE_OBJECT”.

[0129] (3) Temporal difference learning is used to calculate the contribution of action TD, and PageRank is used to evaluate the global influence.

[0130] Extracting key experimental rules:

[0131] Rule 1: If the initial dimensions have been recorded, then dimensional changes can be determined (TD contribution = 0.45, PageRank = 0.48). Rule 2: If the metal has been heated, then high-temperature measurement operations can be performed (TD contribution = 0.39, PageRank = 0.41).

[0132] 3. Rule-enhanced reasoning stage:

[0133] (1) In the test task "determine whether the iron block has thermal expansion characteristics", the system matches the target value through vector retrieval.

[0134] The task area of ​​"material thermal property analysis" was identified, and the most relevant rule set was recalled.

[0135] (2) The LLM rewrites the key rules into natural language to generate prompts that guide reasoning:

[0136] "Please first measure the dimensions of the metal at room temperature and record the data."

[0137] "After the metal is heated, please remeasure and compare the changes in size to determine if there is any expansion."

[0138] (3) The optimized rule prompts are embedded as input into the model reasoning context to guide the agent to plan the experimental process in a reasonable manner, thereby improving the logic of the operation and the reliability of the reasoning.

[0139] Model inference performance:

[0140] Inference Task Dataset

[0141] ALFWorld [1] This dataset focuses on embodied tasks within a home environment, evaluating an agent's exploration, manipulation, and common-sense reasoning abilities in a text-interactive home setting. It includes 134 different tasks covering various home scenarios such as the kitchen, bedroom, and bathroom. Tasks involve object recognition, manipulation, and environmental interaction, such as "put two bars of soap in the trash can" or "put a clean egg in the microwave." Interactions are text-based, requiring the agent to parse environmental feedback and select appropriate actions, such as "open the refrigerator" or "take out the egg." The ALFWorld task has an average of 6 interaction rounds, an action space size of 13, and a context length of approximately 900 tokens.

[0142] Evaluation indicators

[0143] We use the following three evaluation metrics to assess the performance of each method on the test set: Grounding Rate, Progress Rate, and Success Rate:

[0144] GroundingRate measures whether an agent's text generation is a legal action in the current environment, reflecting the agent's cognitive ability and text alignment ability in the task environment. Its definition is as follows:

[0145]

[0146] Where, \(T\) represents the total number of time steps (turns); \(a_t\) represents the action generated by the agent at step \(t\); ​​\(\text{Env}_t\) represents the environment state at step \(t\); ​​(1(IsValid(a t ,Env t ))) is an indicator function. If \(a_t\) is a legal action in the current environment, it takes the value 1; otherwise, it takes the value 0. The GroundingRate ranges from \([0,1]\). A higher value indicates a higher degree of matching between the agent's text output and the environment, indicating a stronger ability of the model to observe, understand, and describe.

[0147] ProgressRate measures the degree to which an agent achieves its objective during task execution, reflecting the progress made in task completion. In this study, progress rate is primarily calculated based on the completion status of sub-objectives. The task progress rate is calculated as follows:

[0148]

[0149] Where \(s_t\) represents the state of the agent at time step \(t\); ​​\(G\) represents the definition of the task objective; \(|G|\) represents the total number of sub-objectives in the task objective; \(|C|\) represents the total number of constraints in the task objective; \(achieved(s_t\) represents the total number of constraints in the task objective; \(achieved(s_t\) represents the state of the agent at time step \(t\)); ... t ,G)\) indicates whether the agent achieves the sub-goal at time step\(t\);\match(s t ,G) indicates whether the agent meets the constraints of the target state at time step\(t\). The value of ProgressRate ranges from \([0,1]\), and a higher value indicates that the agent has made greater progress during task execution.

[0150] SuccessRate evaluates the final success rate of an agent in a task, that is, whether the agent successfully completes the entire task objective. Unlike ProgressRate (which focuses on the gradual progress), SuccessRate only counts as success when the agent fully achieves the task objective; therefore, its calculation method is usually a binary metric. SuccessRate is calculated by whether the final state \(s_T\) satisfies the task objective \(G\), and its definition is as follows:

[0151]

[0152] Where, \(N\) represents the total number of tasks; \(s_T^i\) represents the final state of the \(i\)th task; \(G^i\) represents the target state of the \(i\)th task; It is an indicator function that takes the value 1 when the agent's final state \(s_T^i\) satisfies the task objective, and takes the value 0 otherwise.

[0153] Experimental results

[0154] To compare inference performance, we used zero-shot (0-shot) and one-shot (1-shot) methods to compare with the method in this application (TD-RuAG) to evaluate the impact of different learning paradigms on LLM agents.

[0155] Zero-shot generates answers directly without any examples, relying on the world knowledge and reasoning ability gained during LLM pre-training.

[0156] One-shot provides an example for LLM to understand task format and solution strategies. Compared to zero-shot, one-shot provides a limited task context, enabling LLM to better align with task requirements.

[0157] All comparison methods are based on the Act mechanism. [2] Currently, many module-based proxy frameworks exist to improve proxy performance; however, these frameworks often involve complex designs and are not suitable for all LLMs. Therefore, we chose Act as a unified hinting framework, which requires minimal design and is suitable for most instruction-compliant LLMs. To avoid bias caused by a single base model, we selected three open-source 7B models, and all base comparison methods were compared on these three base models respectively: Mistral-7B-Instruct-v0.2 (Mistral-7B), Meta-Llama-3-8B-Instruct (Llama3-8B), and Qwen2.5-7B-Instruct (Qwen-7B).

[0158] In the ALFWorld task, the agent needs to interact with objects within a partially observable home environment. The task involves multiple sub-objectives, and the environmental state may change as the task progresses. Therefore, the agent must possess strong reasoning and planning abilities. As shown in Table 1, the success rates of the Zero-shot and One-shot methods on this task are only 0.000 to 0.008, indicating that relying solely on LLM pre-training knowledge or hints from a single example is insufficient for the agent to effectively complete the task. In contrast, the TD-RuAG method, combining TD(λ) contribution calculation and rule-based reinforcement learning, achieves success rates of 0.135 and 0.155 for the two base models (except for Mistral-7B, which has a relatively low overall success rate), and a maximum progress rate of 0.387, representing a significant improvement over the comparison methods. Furthermore, TD-RuAG automatically selects high-contribution state-action pairs, enabling the agent to effectively avoid ineffective exploration. Combined with rule-based reinforcement strategies, it dynamically retrieves experience from similar tasks during the reasoning process, thereby optimizing decision quality. Therefore, in the ALFWorld task, TD-RuAG has significant task planning capabilities, which can improve the success rate of multi-step reasoning tasks and effectively cope with the high complexity challenges of tasks in a home environment.

[0159]

[0160]

[0161] Table 1: ALFWorld Inference Results

[0162] Furthermore, such as Figure 3 As shown, this compares the domain rules of TD-RuAG with context learning using only high-quality inference trajectories found during sampling (Demo-AG). TD-RuAG retrieves the rules that contributed most to task progress in historical tasks through a rule extraction strategy and optimizes decisions by combining context learning, while Demo-AG simply uses the trajectory with the highest progress rate found during the sampling process of the large language model as an example without additional rule enhancement. Figure 3As shown, while Demo-AG can improve task progress to some extent, its success rate remains low. For example, Llama3-8B's success rate under example learning (Demo-AG) is only 0.008, indicating that the agent struggles to generalize from a single example to different task instances. In contrast, TD-RuAG combines rule retrieval and contextual learning, enabling the agent to dynamically adjust its policy. For instance, Qwen-7B's progress rate increased to 0.387 and its success rate to 0.135 under TD-RuAG. Demo-AG only provides static examples, while TD-RuAG, through rule augmentation, allows large language models to incorporate experience from multiple tasks during inference, improving the generalization ability of decision-making.

[0163] [1]Wang R, Jansen P, MA, et al. ScienceWorld: Is your Agent Smarter than a 5th Grader? [C] / / Goldberg Y, Kozareva Z, Zhang Y. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, December 2022: 11279–11298.

[0164] [2]Yao S, Zhao J, Yu D, et al. React: Synergizing reasoning and acting in language models [C]. In International Conference on Learning Representations (ICLR), 2023.

Claims

1. A large language model inference method based on temporal difference learning and rule enhancement, characterized in that, Includes the following steps: Step (1) Task trajectory sampling: Based on the base large language model, multiple rounds of random sampling are performed on the training task dataset to generate diverse task inference trajectories; the sampled inference trajectories are filtered, and only valid trajectories are retained; Step (2) Domain knowledge induction: Extract action semantics from the action steps of the effective reasoning trajectory and map them to a unified action space; classify the reasoning trajectory according to the domain to which the task belongs, and aggregate all trajectories in the same task domain to extract the common execution pattern in the domain; A domain knowledge graph is constructed based on the aggregated trajectory set, with deparameterized action semantics as nodes and action semantic dependencies in the trajectory as directed edges, forming a knowledge structure within the task domain; Step (3) Domain rule extraction: The temporal difference method is used to learn the contribution of different actions to the task progress until it converges, and PageRank is combined to evaluate the global influence of actions in the domain knowledge graph. By integrating local contribution and global importance, key task execution rules are extracted and action dependencies are formed, thereby establishing a domain rule set that can be used to optimize LLM reasoning. Step (4) Rule-enhanced large language model inference: For test task data, the most relevant historical tasks are matched based on vector retrieval, and the domain rule set of the corresponding task is obtained; the rules are rewritten in natural language by the large language model itself to make them more interpretable and adaptable. Finally, the optimized rules are integrated into the reasoning prompt text of the large language model to improve reasoning quality and stability; Specifically, step (3) involves using temporal difference learning to calculate the local contribution of each action and combining it with a link analysis-based calculation method to evaluate the global influence of the action on the task graph. Through this process, task execution rules are constructed, and a structured set of domain knowledge is formed. During task execution, the action TD contribution Q needs to be calculated, which is the degree of influence of a certain action on the future task progress under a specific state. The temporal difference error method is used to update the TD contribution Q, thereby ensuring that the Q value can correctly reflect the contribution of the action. Furthermore, qualification traces are used to improve the stability of learning. Its iterative update process is as follows: in, Represents timing difference error, used to measure the current action. Whether the contribution is in line with expectations; It is the immediate contribution, which is the incremental progress of the task directly brought about by the action; It is a discount factor used to control the weight of the influence of future states on current decisions; The value is between [0,1], and the larger one is... This makes the model focus more on long-term returns, while smaller... This makes the model more biased towards short-term gains; It represents the TD contribution level in the next stage, indicating the potential future benefits; It represents the TD contribution of the current action, which is continuously updated during training; It is the qualification trace value, used to record the degree to which a certain action contributes to the progress of the task; It is the learning rate, which controls the step size for updating the Q-value; It is the qualification trace decay factor; A PageRank calculation method is introduced to evaluate the global influence of actions in the domain knowledge graph. The TD contribution and PageRank influence are combined to calculate a comprehensive action contribution, forming task execution rules, which are then stored in the domain knowledge set. in, The final contribution of the representative action, taking into account its local benefits. and global influence , It is a weighting parameter used to adjust the relative weights of TD contribution and PageRank influence; In step (4), when a test task arrives, it is necessary to retrieve the most relevant training task in order to obtain rule knowledge applicable to the current task; for this purpose, a vector retrieval method is adopted, which calculates the description of the current task. With training task description The cosine similarity between them is used to find the most similar training tasks and obtain their rule set; Then, the task rules with the highest contribution are selected from the retrieved task rule set. The rules are used to optimize the reasoning ability of a large language model. The key to this process is to ensure that the large language model only refers to the optimal rules for reasoning optimization, and avoid introducing irrelevant or inefficient rules to interfere with the reasoning results. Finally, the optimized rules are integrated into the LLM's inference hints, enabling it to make more accurate decisions in test tasks.

2. The method according to claim 1, characterized in that, Step (1) Specifically, the training task dataset Each task instance From the mission domain and specific task information Composition, mission domain Indicates the category to which the task belongs; specific task information. It contains the input information for the task; LLM needs to reason about these task instances to generate possible task reasoning trajectories; these trajectories contain the state, actions performed and corresponding task progress at multiple time steps, describing how the large language model advances the task progress during multi-step reasoning. The set of sampling trajectories is formally represented as: in, Represents a large language model in a specific task instance. The complete execution trajectory deduced above includes all decision-making steps from the initial state to the final result; To ensure the diversity and rationality of the sampling trajectories, temperature control and Nucleus sampling were used for trajectory sampling. Furthermore, in the multi-step decision-making process, for each decision step, the agent outputs the current action based on the current environment information; the screening process includes the following key steps: eliminating illegal actions: ensuring that every action in the sampled trajectory conforms to the basic logical rules of the task environment; removing invalid actions: avoiding the generation of redundant inference paths by the large language model.

3. The method according to claim 2, characterized in that, Step (2) specifically maps the actions in the sampled trajectory to a unified semantic space; each decision action From action semantics and motion parameters Composition; among which, action semantics This represents a general operation that spans different task scenarios, while action parameters refer to specific task instances. To aggregate task trajectories within the same domain, action parameters are removed, mapping all actions of the same category to a unified action space, retaining only the action semantics. Then, based on the task domain... Aggregating all trajectories under the current task domain yields And thus build a system for the task domain Directed acyclic graph ; where the node set represents the deparameterized action semantics in the task; the edge set represents the action dependencies in the trajectory; if an action Task execution usually precedes another action. If this happens, then there will be a path in the graph from... arrive The directed edges ( , This reflects the execution logic of actions during the reasoning process.

4. The method according to claim 2, characterized in that, Step (4) Specifically, when a test task arrives, it is necessary to retrieve the most relevant training task in order to obtain rule knowledge applicable to the current task; for this purpose, a vector retrieval method is adopted, which calculates the description of the current task. With training task description The cosine similarity between the training tasks is used to find the most similar training tasks and obtain their rule set. in, This represents a text embedding model used to convert text task descriptions into high-dimensional vector representations, enabling the similarity between different tasks to be measured through mathematical calculations. It is a cosine similarity calculation function used to measure the similarity between two task description vectors; Then, the task rules with the highest contribution are selected from the retrieved task rule set. The rules are used to optimize the reasoning ability of a large language model. The key to this process is to ensure that the large language model only refers to the optimal rules for reasoning optimization, and avoid introducing irrelevant or inefficient rules to interfere with the reasoning results. The filtering rules are calculated as follows: in, It is the mission domain The corresponding complete set of rules; The contribution of an action represents the degree to which an action affects the progress of the task; the contribution calculation combines temporal difference learning and PageRank influence. Indicated according to contribution level Sort them and select the ones that are most helpful for completing the task. Rules; the most relevant set of rules for recall This represents the rule that is most suitable for reference in the current test task; Finally, the optimized rules are integrated into the LLM's inference hints, enabling it to make more accurate decisions in test tasks.

5. The method according to claim 1, characterized in that, It can be applied to automatic planning, intelligent question answering, and embodied intelligence scenarios.

Images